Hot, Warm, And Cold Memory: The Tier Model That Lets Agents Forget Cheaply And Recall Safely

TL;DR

Agents accumulate memory faster than they accumulate revenue. Treating every interaction as equally important produces two failure modes: the bankrupt agent (vector store costs that exceed margin) and the amnesiac agent (aggressive eviction that loses load-bearing facts). The fix is a tier model. Hot memory holds full-fidelity context the agent uses this week. Warm memory holds LLM-compressed summaries the agent might reach for this quarter. Cold memory holds archived raw bytes the agent will only ever touch under audit, dispute, or replay. Each tier has its own latency, cost, retention rule, and access policy. Tier promotion and demotion are first-class events that get logged, audited, and reflected in the agent's behavioral record.

Why Flat Memory Breaks Production Agents

The first agent you ship will store everything. The second agent you ship will store everything because the first one did and you copied the architecture. By the time you have ten agents in production, you have a vector database with a quarter of a billion embeddings, a serverless bill that requires explanation in your monthly board update, and a 3-second tail latency on retrieval that nobody can explain. This is the flat memory problem, and it kills more agent products than hallucination ever has.

Flat memory treats every captured fragment as equal. The user's stated preference from last March sits beside the boilerplate confirmation message they sent yesterday. The transcript of a critical incident response sits beside the transcript of someone asking the agent what time it is. The retrieval system has no priors about what should rank high, what should rank low, what should be searchable at all, or what should have been deleted six months ago. So you index everything, embed everything, retrieve over everything, and pay for everything. Then you tune the retriever, and tune again, and your team starts saying things like "we just need a smarter ranker" because admitting that the underlying data layer is malformed would be admitting you have to rebuild the foundation.

The failure mode that follows is more subtle than cost. It is information drowning. When every retrieval candidate looks plausible, the LLM picks based on surface similarity rather than relevance. The agent answers a question about pricing with a stale fragment from a sales playbook the customer never agreed to. The agent recalls a deprecated tool signature because the deprecation notice was never linked to the original entry. The agent confidently states a fact that was true on the day it was captured and is no longer true today. None of these are model failures. They are memory governance failures. The model did exactly what you would expect a model to do given the context it was handed. The problem is what got handed.

The other thing flat memory destroys is forensic capability. When something goes wrong, you want to know what the agent knew at the moment it made the bad call. With flat memory, you cannot answer that question, because the memory you query today is not the memory the agent had on the day of the incident. The vector store has been written to since. Embeddings have shifted as you re-indexed under a new model. Retrieval ranking has been tuned. The state you care about is gone. The Cortex tier model is not just a cost optimization. It is the precondition for being able to investigate your own product when it misbehaves.

The Three Tiers, And What They Are Each Optimized For

A Cortex-style tier model has exactly three layers, and the discipline is in keeping them three. Two layers cannot express the cost-fidelity trade-off you actually need; four layers create governance complexity that nobody understands by month three. Three is the load-bearing number.

Hot memory is where the agent lives day-to-day. Anything an agent will use this week sits here, in full fidelity, indexed against multiple embedding spaces, and retrievable with sub-100ms latency. Hot memory is small by design. A typical production agent should hold somewhere between five hundred and five thousand entries in hot tier; if you are above that, you have not promoted aggressively enough. Hot memory is expensive per byte and that is fine, because the byte budget is small. The optimization function for hot memory is recall quality, not storage cost. You spend the money to make sure that when the agent reaches for a fact this week, it gets the right one.

Warm memory is the working archive. Anything the agent might use this quarter sits here, but it sits here in a compressed form: an LLM-distilled summary, a structured fact extraction, an indexed claim with provenance and confidence. Warm memory entries are typically one-tenth to one-twentieth the byte size of their hot equivalents. Warm retrieval has a latency budget closer to half a second; you accept this because warm hits are less frequent. The optimization function for warm memory is fact preservation per byte. You will lose nuance in summarization. The discipline is to lose decoration without losing substance, and to instrument the loss so you know when it crosses a threshold that matters.

Cold memory is the legal record. The original raw bytes of every interaction live here, append-only, addressable by content hash, and behind an access policy that requires explicit audit-grade authorization to read. Cold memory is not used for retrieval during normal agent operation. It exists for replay, dispute resolution, regulatory inquiry, and the deeply uncomfortable conversation where a customer says "prove what your agent told my employee on March 14th" and you need to be able to do exactly that. Cold storage is cheap per byte and slow to access; the latency budget is on the order of seconds, and that is appropriate, because nobody is making a routing decision based on a cold fetch.

When To Promote, When To Demote, Who Decides

The discipline that separates a working tier model from a sophisticated mess is the promotion and demotion logic. Tiers without movement are silos. Tiers with chaotic movement are a search-and-destroy mission against your own retrieval quality. The right approach is to make tier transitions first-class, logged events governed by explicit rules.

Promotion from cold to warm happens when an entry becomes relevant to a current task. A pact gets renewed and references an old SLA buried in cold; the SLA is now load-bearing for active work and gets summarized into warm with full provenance pointing back to the cold record. Promotion from warm to hot happens when an entry gets retrieved repeatedly within a sliding window or when a task explicitly pins it. The threshold should be empirical, not aesthetic; we typically see useful behavior when an entry that gets retrieved three times in seven days gets auto-promoted to hot for thirty days. You can tune the constants, but the structure should be retrieval-frequency driven.

Demotion from hot to warm happens on a calendar boundary, not a usage one. An entry that has not been retrieved in fourteen days demotes regardless of how recently it was created. This is the rule that prevents your hot tier from filling with novel-but-unused entries that the agent embedded in a moment of enthusiasm and never returned to. Demotion from warm to cold is more conservative; we suggest sixty to ninety days of dormancy before warm content gets archived to cold. The reason for the asymmetry is that warm summaries are cheap to keep around and expensive to regenerate from cold, so you want to err on the side of holding them.

The decision authority matters as much as the rules. Promotion and demotion should run as a continuous background process, owned by the memory subsystem, with the agent itself unable to forcibly promote arbitrary entries. The agent can request a promotion ("this entry is critical for this task, please pin it") but the memory governor decides whether to honor the request and logs the request either way. This separation is what prevents an agent from building a runaway hot tier by self-pinning everything it touches.

Reader Artifact: The Tier Promotion And Demotion Decision Table

Use this table as a starting point for memory tier policy. The thresholds are tuneable by domain; the rule shape is not.

The most common configuration error is letting agents pin without expiry. The second most common is letting demotion run on creation date instead of retrieval recency. The third is treating cold as cheap-warm rather than as a legal record with separate access controls. Avoid all three.

Retrieval Latency By Tier, And Why The User Should Never Notice

The latency budgets above are not arbitrary. They reflect the structure of how an agent should reach for memory during a turn. A single agent turn should never block on more than one hot retrieval and one warm retrieval, sequentially. Cold retrievals never happen on the critical path of a turn; they happen on audit, on replay, or on explicit user-initiated dispute flows. If your architecture requires a cold fetch to answer a normal user question, you have placed something in cold that should be in warm.

The failure mode of getting this wrong is tail latency. Your p50 turn time looks fine because most turns hit hot and never go further. Your p99 turn time is awful because some turns reach into cold and take three seconds. Users do not perceive average latency; they perceive worst latency, because the bad turns are the ones they remember. A tier model with disciplined budgets gives you a p99 you can defend.

There is also a cost angle that is worth being explicit about. The cost-per-byte ratios for the three tiers, in our experience, are roughly one hundred to ten to one. Hot memory, with its multi-index embeddings and replication and low-latency requirements, costs around one hundred units per byte stored. Warm costs around ten. Cold costs around one. This means the economic incentive to demote aggressively is real and large. An agent with one million entries in hot tier costs roughly the same as an agent with one million entries split ten percent hot, thirty percent warm, sixty percent cold. The difference is whether you can afford to scale to ten such agents or whether you cannot.

The Compression Step, Where Most Programs Quietly Break

The transition from hot to warm is the hinge of the whole system. Hot entries are full-fidelity transcripts; warm entries are LLM-compressed summaries. The compression step is where you trade fidelity for cost, and if you do it carelessly, you trade away facts you needed.

The wrong way to compress is to feed the raw entry to a model with the prompt "summarize this in 200 words." You will get back fluent prose that drops the dollar amount, the date, the customer ID, the SLA threshold, and the specific tool name that was called. The summary will read well. It will be useless when the agent reaches for it three months later trying to remember what was actually agreed.

The right way to compress is structured: extract a schema of facts (who, what, when, how much, what tool, what outcome, what confidence) and write the summary as a constrained narrative anchored on those facts. The summary is then verifiable against the raw entry by extracting the same schema again from the raw and diffing. If the diff is empty, the summary is fact-faithful. If the diff is non-empty, the summary lost something and you either re-compress or you keep the entry hot longer.

This is the topic of a separate piece, but the short version is: compression without a fact-diff verification step is summarization theater. You feel like you have a working memory archive. You actually have a beautifully written archive of the wrong things.

How Tier Movement Feeds The Agent's Behavioral Record

A pact-governed agent under composite scoring has memory dimensions in its score. This is not just memory hygiene; it is reputation-affecting behavior. When an agent prematurely demotes an entry that turns out to be load-bearing, the resulting failure is logged against the memory dimension. When an agent over-pins to hot and runs up cost without recall benefit, that pattern shows up in the cost-efficiency dimension. When an agent's compression step loses facts that the agent later confabulates around, the resulting hallucination is traceable to the compression event.

The consequence is that memory tier policy is not an ops concern. It is a behavioral concern, and it lives in the agent's pact. The pact says: "this agent will retain operational context for 90 days at warm fidelity, archive raw to cold for 365 days, and not pin more than 50 entries to hot at any time." Violations of the pact are flagged. Compliance shows up in the score. This is what makes a memory architecture trustworthy: not that it is technically clever, but that the agent's adherence to it is observable and consequential.

Multi-Tenancy And The Per-Tenant Tier Budget Problem

When your agent serves a single customer, the tier model is straightforward: one budget, one set of policies, one substrate. When your agent platform serves hundreds or thousands of tenants, the tier model becomes a multi-dimensional optimization problem that most teams underestimate by an order of magnitude. Each tenant has their own usage pattern, their own retention requirements, their own willingness to pay for hot-tier latency. A small tenant whose entire memory store fits in five hundred entries should not pay the per-byte rate of a large tenant whose store demands aggressive demotion. A regulated tenant whose retention obligations require seven-year cold storage should not subsidize an unregulated tenant whose data has a thirty-day TTL.

The naive approach is one tier policy for the whole platform. This produces predictable failure: small tenants get bills that exceed their value, large tenants experience retrieval latency they cannot afford to tolerate, regulated tenants discover their data was demoted in ways their compliance officer cannot accept. The platform either loses small tenants to cost, large tenants to performance, or regulated tenants to liability. Often all three simultaneously.

The correct approach is per-tenant tier policy with shared substrate. The substrate is one infrastructure (shared vector indexes, shared cold archive, shared compression workers) but each tenant has its own promotion thresholds, demotion windows, retention rules, and cost budgets. The shared substrate amortizes infrastructure cost across the tenant base; the per-tenant policy lets each tenant have memory governance appropriate to their needs. The implementation is a tier policy registry keyed by tenantId, consulted by the substrate at every decision: this entry belongs to tenant X, X's policy says hot retention is 21 days not 14, retention windows extended for regulatory compliance, cold archive in EU region for data residency.

The operational complexity is real but bounded. Most platforms can converge to three or four tier-policy templates that cover the bulk of tenants (free tier, standard tier, regulated tier, enterprise tier with custom policy) with the long tail handled through explicit per-tenant overrides. The discipline is to expose the tier policy as a first-class tenant configuration surface, with monitoring that shows each tenant their per-tier byte counts, retrieval hit rates, and projected costs. Tenants who can see their tier behavior can tune their own policies; tenants who cannot will eventually surprise you with a support ticket about their bill.

The Embedding Architecture Across Tiers

Hot, warm, and cold are not just storage tiers; they are also embedding tiers, and the embedding strategy needs to vary by tier in ways most implementations miss. Hot tier benefits from multiple embedding spaces: a strong general-purpose embedding for semantic retrieval, a domain-specific embedding fine-tuned on your corpus for higher-precision matching, and potentially a code or structured-data embedding if your domain includes those. Multi-embedding hot retrieval costs more per query but produces measurably better recall on the small set of entries that make up the hot tier. The cost is bearable because the hot tier is small.

Warm tier should use a single embedding, ideally the strongest general-purpose embedding available, generated fresh from the warm summary text. Critically, the warm embedding is a different artifact from the hot embedding; you do not carry the hot embedding into warm because the underlying content has been compressed and the original embedding no longer accurately represents the warm summary. Generating a fresh embedding on demotion is part of the demotion cost, and you should budget for it.

Cold tier should typically not be embedded at all. Cold is for replay, audit, and dispute resolution; it is not part of the live retrieval path. Embedding cold tier means paying ongoing index maintenance cost for content the agent never actually reaches for, and inviting the temptation to start using cold for retrieval because you already paid to embed it. Resist this. The discipline that keeps cold tier cheap is the discipline that lets you afford to keep things in cold for the legally-mandated retention windows.

The asymmetry across tiers (multi-embedding hot, single-embedding warm, no-embedding cold) is not a clever optimization; it is the structure that lets the cost-per-byte ratios match the access patterns. Get the embedding strategy wrong and your tier model loses its economic justification; you end up paying for indexing capacity you do not need on tiers that should be cheap, while underinvesting in the index capacity you do need on the tier that is hot for a reason.

The Failure Mode Of Tier Policy That Nobody Owns

Tier policy decays without an owner. The retention windows you set at launch are not the retention windows your business needs eighteen months in. The promotion thresholds you tuned for one usage pattern are wrong when the usage pattern shifts. The compression schemas drift as new domains get added. Without a designated owner who reviews the policy quarterly and tunes it against observed behavior, the tier model becomes an artifact of historical decisions that nobody can defend.

This is the same failure mode you see in any infrastructure system that lacks an owner: AWS accounts where the bill keeps growing because nobody is responsible for cost; database schemas where every team adds columns and nobody removes them; Redis instances where the eviction policy is set to whatever was the default in 2019. The tier model is not different. It needs an owner.

The owner's job is not to author policy from scratch. The owner's job is to look at the actual data: hit rates by tier, cost by tier, compression failure rates by domain, retrieval latency distributions, retention compliance audit results. The data tells you which thresholds are wrong. Hit rate at hot tier is 95%? You can probably demote more aggressively without losing recall quality. Compression failure rate is 20% in the customer-service domain? Your fact schema is missing important fields. Cold retrieval volume is climbing? Your warm retention is too short and you are paying audit-grade access costs for routine queries.

The owner produces a quarterly report that says: here is what we changed, here is why, here is what we expect the change to do, here is how we will measure whether it worked. This is the same review cadence you apply to other infrastructure decisions. The reason most platforms do not apply it to memory tier policy is that nobody designated the owner. Designate the owner. Run the cadence. Your memory infrastructure will pay you back many times over the cost of the meeting.

Counter-Argument: Just Use A Bigger Context Window

The honest counter-argument is that frontier models keep getting larger context windows, and at some point you should just shove everything into context and let the model sort it out. There is something to this. Long-context models genuinely reduce the engineering burden of retrieval-augmented architectures. If your agent is small-scale, your interactions are short, and your cost tolerance is high, the long-context approach can carry you further than people give it credit for.

It does not carry you to production. The economics break first. Loading a million tokens of context for every turn is, even at heavily discounted prompt-caching rates, an expensive way to answer a question that needs three facts. The latency breaks second. Even the best long-context models have observable degradation in needle-in-a-haystack accuracy past a certain depth, and they have noticeable first-token latency penalties at extreme context sizes. The forensics break third. A long-context architecture has no story for what the agent knew on the day it failed, because there was no curated memory store; there was just a context window that got rebuilt from raw sources every turn.

The right way to think about this is that long context replaces some retrieval, not all memory. Tiered memory plus selective long-context is the architecture that scales. Long-context as a substitute for memory governance is the architecture that scales until your finance team sees the bill.

Pre-Production Calibration: How To Tune Thresholds Without Production Data

The most common question from teams adopting a tier model is: how do I pick the thresholds when I have no production data yet? The fourteen-day demotion threshold and the three-retrievals-in-seven-days promotion threshold sound reasonable, but they are starting points calibrated against observed usage patterns from production fleets. New deployments need their own calibration approach.

The right pre-production calibration starts with synthetic load that approximates the expected access pattern. If your agent will handle customer service tickets with an expected distribution of repeat-customer queries (most customers contact you once, some contact you many times over months), you can simulate that distribution. Generate synthetic interactions, run them through the agent and the substrate, and observe what fraction of memory entries get re-retrieved within various windows. The empirical access curve from the simulation tells you where to set your initial thresholds.

The second source of calibration is your existing data, if you have any. Even if you have not deployed an agent, you likely have customer interaction histories somewhere: support tickets, sales call notes, account activity logs. Inspect the temporal access patterns. How often is data from six months ago referenced in current activity? How often is data from six weeks ago referenced? The answers shape your warm-to-cold demotion timing. If six-month-old data is referenced in 10% of cases, six months is too aggressive for cold demotion. If it is referenced in 0.1% of cases, six weeks might be appropriate.

The third source is industry baselines. Customer service agents typically see retrieval distributions where 80% of retrievals hit the most recent two weeks of memory and 95% hit the most recent two months. Compliance review agents see different distributions: queries reach back years routinely. Trading agents see almost no retrieval beyond the current trading day. Your domain has a baseline; find one or two agents in your domain and ask their operators what their access curves look like.

The fourth source is intentional initial conservatism. When in doubt, start with longer hot retention and shorter warm retention. You can tighten hot later as you observe what is actually used. You cannot recover entries that demoted to warm and then demoted to cold and then deleted under retention policy. Conservative initial thresholds let you collect the empirical data you need without losing the data you want to keep. Calibration is iterative; the goal of pre-production tuning is to get close enough that the first quarter of production data lets you converge to the right values, not to nail it on day one.

What Armalo Does

The Cortex memory subsystem implements the hot, warm, and cold tier model with explicit promotion and demotion rules. New writes start in hot. A background governor demotes hot entries that have not been retrieved in 14 days into warm via fact-faithful compression with a diff-verification step. Warm entries that have not been retrieved in 60 to 90 days demote to cold with the warm summary deleted and the raw record retained for the configured retention window. Cold reads do not auto-promote and require audit-grade authorization. Tier transitions are first-class events written to the agent's behavioral record. Memory dimensions in the composite score reflect tier discipline: agents that pin without justification, fail to demote, or compress lossily see those behaviors land in their score. Pacts can encode tier policy directly, so an agent's memory behavior is not just an ops setting but a contractual commitment that counterparties can verify through the trust oracle.

FAQ

Q: Why three tiers and not five? A: Three tiers express the load-bearing trade-off (operational vs. archival vs. legal) without creating governance complexity that operators cannot reason about. Five-tier systems exist; they are usually two real tiers and three vanity tiers that nobody can describe the policy for.

Q: Should tiers be per-agent or shared across agents? A: The tier policy is per-agent. The underlying storage substrate can be shared. An agent's memory budget, retention windows, and pin allowances are part of its configuration and its pact, even if the bytes live in a shared vector store.

Q: How do you handle tier transitions for memory the agent disagrees with? A: Demotion is a memory subsystem decision, not an agent decision. An agent can request that an entry be retained at hot tier with justification; the request is logged. The governor honors the request up to a budget cap and then refuses, also logged.

Q: What happens to embeddings when an entry demotes? A: Hot embeddings are typically deleted on demotion to warm; warm has a separate, cheaper embedding strategy. Embeddings on cold are typically not maintained at all because cold is not used for similarity retrieval.

Q: Does compression to warm lose information? A: Yes, by design. The discipline is to lose decoration and not facts. The fact-diff verification step (extract structured facts from raw, extract from summary, diff) catches lossy compression and triggers either re-compression or extended hot retention.

Q: How does this interact with regulatory retention? A: Cold tier holds the raw record for as long as your retention policy requires and not a day longer. Tier transitions are audit events; deletion at retention end produces a tombstone with content hash so the absence of the data is itself provable.

Q: What if an agent only ever retrieves recent stuff? A: Then your warm and cold tiers are doing their job, which is being available without being expensive. Most production agents have heavy recency bias in retrieval, and the tier model is designed for exactly that distribution.

Q: How do you measure whether the tier policy is working? A: Three signals: hit rate by tier (what fraction of retrievals are satisfied at hot, warm, cold respectively), cost per resolved task, and false-confabulation rate after compression. If hot hit rate is low, you are demoting too aggressively. If cost is high, you are not demoting enough. If confabulation rate climbs after compression events, your fact-diff is not catching what it should.

The Migration Path: How To Move A Flat-Memory Agent To Tiered Memory

Most teams reading this already have agents in production with flat memory. The question is not whether to build tiering; the question is how to migrate without breaking what works. The migration has four phases and the discipline is to do them in order.

Phase one is observability. Before you change any architecture, instrument the existing flat memory store to produce the data you would need to make tiering decisions. Track per-entry retrieval frequency over a rolling window. Track per-entry age. Track which entries are returned by which kinds of queries. Run this for at least four weeks; ideally eight. The output is a baseline that tells you what your existing access pattern actually is, not what you assume it is.

Phase two is shadow tiering. Implement the tier model as a parallel substrate that mirrors the flat store but does not yet serve queries. Run the same retrievals against both substrates; compare results. Where the tiered substrate returns different rankings or different content, investigate. Most differences will be cases where the tier model correctly identified that an entry should not have been promoted but the flat store returned it anyway because there was nothing else to return. A few differences will be cases where the tier model demoted too aggressively; tune the thresholds.

Phase three is gradual cutover. Route a small percentage of queries (start with 5%) to the tiered substrate as the source of truth, while continuing to write to both substrates. Monitor agent behavior on the tiered-served traffic for any degradation in answer quality, latency, or downstream task success. If degradation appears, identify the cause (usually compression failure or wrong tier assignment), fix it, and increase the cutover percentage. If no degradation appears, increase the percentage on a schedule (10%, 25%, 50%, 100%) over two to four weeks.

Phase four is decommissioning the flat store. Once 100% of queries are served from the tiered substrate and you have run that way for a full retention cycle of the longest tier, you can stop writing to the flat store. Keep the flat store as a backstop for a few months; then archive it to cold-equivalent storage; then delete it according to your retention policy. Do not skip the backstop period. Migrations that look complete sometimes turn out to have missed cases that only appear under specific load patterns.

The whole migration takes three to six months for a typical production agent platform. The discipline is to not rush. A migration that breaks production retrieval is worse than no migration at all, because it destroys both the flat store you had and the tiered store you were building. Slow, observable, reversible cutover is the only safe path.

Bottom Line

Memory is not a database problem. It is a governance problem with a database substrate. A flat memory architecture lets your costs run away, your retrieval quality decay, and your forensic capability evaporate. A tiered architecture with disciplined promotion, demotion, fact-faithful compression, and audit-grade cold storage gives you an agent that can remember what matters, forget what does not, and explain itself when something goes wrong. The discipline is not in the storage layer. It is in the rules that move data between layers, the events that get logged when data moves, and the pact that ties the memory behavior to the agent's reputation. Build the tier model first, before you build the agent on top of it, and you will not have to apologize to your CFO or your customers later.