The Memory Audit: A Quarterly Practice That Catches Slow Failures Before They Compound

TL;DR

Memory failures in production agents are rarely sudden. They drift in over weeks and months: the fraction of entries with verifiable provenance declines two percentage points per quarter, attestations age past their freshness window without renewal, retrieval boundaries get loosened by well-meaning patches, key facts develop subtle inaccuracies that compound through the consolidation pipeline. Each individual drift is small enough to be invisible at the per-week level; the cumulative effect over a year is catastrophic. The quarterly memory audit is a structured practice for catching these drifts before they compound. This post defines the four audit dimensions, walks through how to instrument each one, names the thresholds that should trigger action, and provides a Quarterly Memory Audit Worksheet you can run against any production memory store.

Intro: The agent that had been wrong for nine months and nobody knew

A legal services platform we worked with ran an agent that helped paralegals draft routine filings. The agent had been live for two years; it had a memory store of about 800,000 entries covering case law citations, jurisdiction-specific filing requirements, court schedules, and procedural quirks. The agent's outputs were spot-checked by senior paralegals at a sampling rate of roughly two percent; the spot-check error rate was steady at around 1.5%, which the team considered acceptable.

A new senior paralegal joined and started spot-checking at a slightly higher rate. She noticed that a specific kind of error (citing a procedural rule from one jurisdiction while drafting for another) was appearing more often than the others. She traced the pattern back through the agent's outputs and found that the error had started appearing roughly nine months earlier and had grown gradually in frequency. The cumulative count of affected filings was in the thousands. The cause turned out to be a memory entry that had been incorrectly tagged for jurisdiction during a tagging-schema migration; the agent's retrieval logic was surfacing the entry for queries that should have been filtering it out, because the tag mismatch had silently broken the filter. The fix was small. The cleanup of nine months of affected filings was enormous.

The team's post-incident review identified the root cause as the absence of any periodic audit of the memory store. The spot-checks of agent outputs had caught individual errors but had not surfaced the pattern that pointed back to the underlying memory issue. There was no recurring practice that would have looked at the memory store and asked, structurally, 'is everything still configured the way it should be?' The team built such a practice in the wake of the incident; the practice has caught three further issues in the year since, each smaller than the original because the audit window is shorter.

This pattern is universal. Memory stores degrade in ways that are invisible to per-output monitoring because the output looks fine; only the structural audit catches the drift. The rest of this piece argues for treating memory audit as a quarterly operational practice, defines the four dimensions worth auditing, and provides the worksheet that turns the practice into something repeatable.

Why output monitoring is not enough

Most teams that run agents in production have some form of output monitoring: a sample of outputs is reviewed by humans, scored on quality, and the aggregate quality is tracked over time. This is necessary but insufficient. Output monitoring catches the symptoms of memory failure, not the failures themselves.

The gap is structural. An output is a sample of the agent's behavior at a moment; the memory store is the substrate that produces the behavior. A subtle memory drift produces outputs that are individually within the normal quality distribution but collectively shifted in a specific direction. The shift is invisible at the per-output level (each output looks fine on its own merits), and it shows up in the aggregate only after enough outputs accumulate to make the shift statistically detectable. By then, the damage has been done.

There are four specific failure modes where output monitoring is provably insufficient. First, a memory entry that is wrong but not retrieved often: the agent's outputs are mostly correct (because the bad entry rarely fires), so the per-output quality is high, but the few outputs that do retrieve the entry are silently wrong. Second, a tag or filter regression that loosens retrieval boundaries: the agent's outputs cite entries it should not have access to, but the citations look plausible, and the human reviewer (who does not have the access policy in mind) does not flag them. Third, attestation expiration on entries that are still being retrieved: the entries are technically still in memory and producing outputs, but the attestations that authorized them have expired, and downstream counterparties relying on the attestations are getting stale evidence. Fourth, gradual provenance erosion: the fraction of entries with fully verifiable provenance declines as the consolidation pipeline produces self-derived entries that lose their upstream connection; the outputs look fine, but the structural verifiability of the underlying reasoning has degraded.

None of these are caught by output monitoring at any reasonable sampling rate. They require auditing the memory store itself, against structural properties that the store should satisfy, on a recurring schedule. The audit is the complementary practice; output monitoring catches what the audit does not, and the audit catches what output monitoring does not.

Audit dimension one: provenance coverage

Provenance coverage is the first dimension because it is the foundation of every other property worth verifying. Provenance is the metadata trail that records where a memory entry came from: the originating source, the timestamp, the agent or user who introduced it, the operation that produced it. Without provenance, no other audit dimension is meaningfully checkable.

The audit measures provenance coverage along three sub-dimensions. The first is presence: what fraction of memory entries have any provenance at all? Healthy stores keep this above 99%; entries with no provenance should be rare and explicable (typically infrastructure entries or seed data with documented exception status). A presence rate below 95% is a red flag; below 90% is a structural failure that needs immediate attention.

The second sub-dimension is verifiability: of the entries with provenance, what fraction can be verified against an upstream source? An entry whose provenance points to a deleted upstream, an unreachable URL, or an unverifiable claim does not pass verifiability. The audit attempts a sampled verification across a percentage of provenance-bearing entries (we sample two to five percent depending on store size) and reports the verification success rate. Healthy stores keep this above 90%; below 80% indicates that the upstream sources are degrading faster than the memory store is being maintained, and the gap will continue to widen without intervention.

The third sub-dimension is trail integrity: do the provenance trails resist tampering? The audit verifies the cryptographic signatures on a sample of provenance trails and reports the signature verification rate. Healthy stores keep this at 100%; any failure indicates either a key rotation that was not propagated correctly or a tampering event that needs investigation.

The combined provenance coverage report is a single quarterly snapshot that the operator can compare against prior quarters. The trend matters as much as the absolute number. A store with 92% verifiability that was 95% last quarter is degrading and needs intervention; a store with 88% verifiability that was 85% last quarter is improving and the trajectory is correct. Most operators find that the first audit produces a worse number than they expected; the second audit, after deliberate cleanup, produces a much better number; subsequent audits maintain or improve the baseline.

Audit dimension two: attestation freshness

Attestation is the signed metadata that records authorized lifecycle events on memory entries: ingestion, transfer, modification, lending, recovery. Attestations have effective dates and, sometimes, expiration windows; an attestation that has aged past its window is no longer authoritative even though the entry it authorized may still be in active use.

The audit measures attestation freshness along two sub-dimensions. The first is coverage: what fraction of entries that should have attestations actually have them? Some entry classes (those subject to lending agreements, those tied to active pacts, those with regulatory significance) require attestations as a structural property; entries in these classes without attestations are violations of the operator's own policy. The audit enumerates the policy-required attestation classes and reports per-class coverage.

The second sub-dimension is currency: of the attestations that exist, what fraction are still within their effective window? An attestation that expired three months ago is no longer authorizing anything; an entry whose authorization has expired is operating in a quiet violation of the original consent. The audit reports the fraction of attestations within window, the fraction expired but not renewed, and the fraction expiring in the next quarter (the leading indicator that informs renewal planning).

A particular pattern worth watching is the silent re-use of expired attestations. An entry whose attestation expired but whose retrieval logic does not check the expiration window will continue to surface the entry to queries; downstream consumers relying on the attestation for verification will get stale evidence without warning. The audit specifically probes for this pattern by sampling retrievals and verifying that any entries returned have current attestations. The probe should produce a 100% success rate; any failure indicates that the retrieval logic is not enforcing the freshness check.

Attestation renewal is itself an operational practice that benefits from the audit. The audit's report of attestations expiring in the next quarter is the input to the renewal queue; the renewal queue is worked through systematically before the expirations actually fire. This converts attestation freshness from a reactive concern (something failed, fix it) to a proactive one (something is about to expire, renew it). The shift in posture is important; expired attestations are much harder to reason about retroactively than upcoming ones.

Audit dimension three: retrieval boundary compliance

Retrieval boundaries are the rules that define which queries can retrieve which entries: tenant isolation in multi-tenant stores, sensitivity class filters, lending permission scopes, attestation-class restrictions. The boundaries are enforced by the retrieval logic; the audit verifies that the enforcement is actually working.

The audit takes the form of an adversarial probe. A test suite of carefully constructed queries attempts to retrieve entries that should be out of bounds for those queries; every retrieval that should fail is checked, and every retrieval that should succeed (the negative control) is also checked. The probe success metric is the fraction of attempted boundary violations that were correctly refused. Healthy stores produce 100% on this metric; any failure is a structural defect that needs immediate fix.

The probe suite needs to evolve with the store. As new boundary types are added, new probes are added. As the schema of metadata fields changes, probes that depend on those fields need to be updated. The probe suite is essentially a regression test for the retrieval logic, and it should be maintained with the same discipline as any other test suite. Many teams treat the probe as a one-time exercise; this is a mistake. The probe is most valuable as a recurring check that keeps the retrieval logic honest as the codebase evolves.

A second probe within this dimension is the access pattern audit: what fraction of retrievals in the past quarter actually exercised boundary checks? An entry retrieval that did not have to check any boundary is fine; one that should have checked a boundary but did not is a defect. The audit instruments the retrieval path with boundary-check counters and reports the per-query-class boundary check rate. Healthy stores produce a rate near 100% for query classes where boundaries should apply; lower rates indicate code paths that are bypassing the boundary check, which is a finding worth investigating.

The third probe is the silent-success check. A retrieval that returns no results when results should have been returned is a quiet failure mode; it does not produce an error, the agent reasons over an empty result set, and the downstream output may be wrong without anything obviously broken. The audit samples queries with known-good answers, runs them through the live retrieval path, and verifies that the expected entries are surfaced. Failures here usually point to recently introduced filters that are too aggressive, or to indexing issues that have caused entries to drop out of retrieval reach.

Audit dimension four: key fact drift

Key fact drift is the most subtle of the four audit dimensions and the one most likely to produce real-world incidents like the legal services example in the introduction. Memory entries that capture specific facts about the world (regulatory rules, system configurations, customer-specific preferences, technical specifications) need to remain accurate as the world changes. The audit verifies that the entries the agent is actively retrieving still match the upstream truth.

The audit defines a set of key facts: entries whose accuracy is load-bearing for the agent's operation, that are observable against an external upstream, and that are at risk of drift over time. The set is workload-specific; a customer service agent might define key facts as 'currently published product specifications and pricing,' a legal services agent as 'currently effective procedural rules per jurisdiction,' a technical support agent as 'currently supported software versions and their associated configurations.' The set should be small enough to verify thoroughly (typically dozens to low hundreds) and significant enough that drift on any of them is operationally meaningful.

The verification step compares each key fact against its current upstream value. The comparison can be automated for upstream sources that are queryable (a public API, a published document, a reachable database), and manual for upstream sources that are not. The audit reports the per-fact match rate and produces a detailed report of any mismatches, including the divergence between what the memory says and what the upstream says.

The drift rate per quarter is itself a useful metric. A store whose key facts drift at 1% per quarter is well-maintained; a store whose key facts drift at 5% per quarter is being out-paced by the upstream changes and needs more frequent reconciliation. The drift rate should be tracked over time; a sudden spike in drift indicates that a major upstream change has happened and the memory store has not yet absorbed it.

The remediation for drift is workload-specific. Sometimes the right response is to update the memory entry to match the upstream; sometimes it is to expire the entry and force the agent to re-derive the fact from the upstream on next retrieval; sometimes it is to flag the entry for human review because the upstream change is significant enough to require operator judgment. The audit report should distinguish among these recommendations rather than producing a generic 'this entry has drifted' alert.

Operational mechanics: who runs the audit and when

The audit needs an owner. In small teams, the owner is the engineer who maintains the memory infrastructure; in larger teams, the audit is owned by a memory governance role distinct from the agent operators. The key point is that the owner is responsible for running the audit, producing the report, surfacing findings to the right downstream consumers, and tracking remediation through to closure.

The audit calendar should be set at the start of the year and not adjusted casually. We use a quarterly cadence (one audit per quarter, on the second week of the quarter) for most workloads. High-velocity workloads benefit from monthly cadence on the most failure-prone dimensions (typically retrieval boundary compliance). Low-velocity workloads can run semi-annually if the volume of change is genuinely low. Whatever cadence is set, the calendar should be respected; rescheduling becomes endemic if exceptions are made, and the audit becomes a discipline-in-name-only.

The audit report should follow a stable format quarter over quarter. The format should include: per-dimension metric snapshots with comparison to prior quarters, list of findings with severity levels, list of recommended remediations with owners and due dates, list of remediations from prior quarters that have been completed, list of remediations that are overdue. The stability of the format makes the trend visible; reports that change format every quarter make trend analysis harder than it needs to be.

Findings should be triaged into severity levels. A typical scheme: critical (memory store is producing actively wrong outputs and remediation is needed immediately), high (drift is significant and remediation is needed within the quarter), medium (drift is detectable but operationally tolerable; remediation can be scheduled), low (the audit found something to note but no action is required). Triage should be done at the time of the audit, with the triage author named in the report so future audits can see who made which calls.

The audit's findings should also feed the broader operational practice. A high-severity finding about provenance coverage should produce a follow-on review of the ingestion pipeline; a recurring medium-severity finding about key fact drift should prompt a discussion about whether the workload's reconciliation cadence needs to change. The audit is most valuable when it is not a self-contained ritual but a feeder of insights into other operational practices.

Counter-argument: 'We do not have the bandwidth for quarterly audit'

The most common pushback on the quarterly audit cadence is that the team does not have the bandwidth for it. The argument: the team is small, the operational pressure is constant, every hour spent on audit is an hour not spent shipping, and the audit's findings (when there are findings) are mostly small enough to address opportunistically.

The argument is sympathetic but mostly wrong. The audit's bandwidth cost is not what people estimate; once the instrumentation is in place, a single quarterly audit takes one engineer one to three days, depending on store size. That is a small fraction of the engineering capacity of even a small team; if the team genuinely cannot find one engineer-week per quarter for memory hygiene, the team's overall operational discipline is in worse shape than the audit itself can address.

The audit's findings, when there are findings, are often not small. The legal services example in the introduction was 'a tagging issue that took a few hours to fix,' but the cleanup was nine months of mis-filed paperwork. The right framing is not 'how big is the issue' but 'how late are we in finding it.' A quarterly cadence catches issues at most three months late; an annual cadence catches them at most a year late; no cadence at all catches them whenever the symptoms are large enough to be impossible to ignore, which is much later.

The rebuttal to the bandwidth argument that we have found most persuasive in practice is to invert the framing. The team does not have the bandwidth to skip the audit, because the absence of audit produces incidents whose response cost dwarfs the audit cost by orders of magnitude. The audit is the cheap option; skipping it is the expensive option, just on a different schedule. Reframing it this way usually unlocks the bandwidth that was nominally unavailable.

The one case where the audit cadence can be relaxed honestly is for very small or very low-stakes deployments. A single agent serving a single internal team, with a memory store under fifty thousand entries and no external counterparties, can run on annual cadence (or even ad hoc cadence) without much risk. The cadence should always be deliberate; the question is what the right cadence is, not whether to skip altogether.

What Armalo does

Armalo's Cortex memory layer (hot, warm, cold tiers) is instrumented for all four audit dimensions out of the box. Provenance coverage, attestation freshness, retrieval boundary compliance, and key fact drift are computed continuously and surface as quarterly audit reports against any operator's memory store. Operators see a per-quarter snapshot, prior-quarter comparisons, and a findings list with severity triage; remediation tracking is built in.

The audit emits attested events that become evidence for the multi-LLM jury (with top and bottom 20% outlier trimming) when memory hygiene becomes relevant to a pact compliance review. Composite trust scores incorporate the agent's audit history as a contributing signal in the safety dimension; consistent audit practice and timely remediation of findings contribute positively, while skipped audits and chronic overdue remediations contribute negatively. The score decay function (one point per week) ensures that one-time audit failures do not permanently mark an agent, while patterns of neglect compound naturally.

For operators hosting agents on behalf of others, the audit infrastructure supports per-tenant audits with results exposed to the tenant alongside the operator. Tenants reading the trust oracle at /api/v1/trust/ can see an agent's certification tier (Bronze, Silver, Gold, Platinum) and the agent's audit hygiene as part of the same query, supporting informed decisions about whether to engage with the agent. Where USDC-denominated escrows on Base L2 are configured to backstop pact obligations, audit findings that bear on the obligations can trigger structured reviews that affect settlement.

FAQ

Can the audit be fully automated?

Mostly. The metric collection, the boundary probes, the freshness checks, and the key fact verification (for upstream sources that are queryable) can all be automated. Triage of findings and judgment about remediations need human attention. The right split is automation for the data-gathering and human attention for the interpretation and decisions.

What if the audit finds something the operator cannot fix?

Findings that cannot be fixed in the current quarter should be carried forward into the next quarter's report with explanatory notes. A finding that persists across multiple quarters without remediation is itself a signal worth surfacing; the audit owner should escalate persistent findings to the operator's leadership as a separate operational issue.

How does the audit interact with the memory triage in the prior post?

The audit catches drift in the memory store; the triage decides what to keep, compress, archive, or delete. The two practices feed each other: audit findings about declining utility on certain cohorts inform the next triage's decisions, and triage decisions about retention inform the next audit's expectations of what should still be present.

What is the right severity threshold for surfacing findings to leadership?

Critical findings should be surfaced immediately, not waiting for the audit cycle. High findings should be summarized in the quarterly report sent to leadership. Medium and low findings should be tracked internally without escalation. The exact thresholds should be calibrated to the operator's organizational appetite for visibility.

How do I know my key fact set is the right set?

The key fact set is right when audits against it consistently catch real drift before the drift produces incidents. If the set is producing too few findings to justify the audit effort, it is too narrow. If audits keep producing findings on entries that turn out not to matter, the set is too broad. The set should evolve based on what the audit actually catches versus what real incidents reveal.

What happens to the audit if the memory schema changes?

A memory schema change is a major event for the audit. The probes that depend on schema fields need to be updated; the metric definitions may need to change; the historical comparison may have to be discontinued and restarted. Plan for the audit migration as part of the schema migration, not as an afterthought.

Does the audit work for read-only memory stores (knowledge bases that do not accept new entries)?

Yes, with adjustments. Provenance coverage and attestation freshness are still relevant; retrieval boundary compliance is still relevant; key fact drift is even more relevant because read-only stores cannot easily incorporate upstream changes. The audit cadence on a stable read-only store can usually be longer than on an actively growing store.

Bottom line

Memory failures in production agents drift in slowly and become catastrophic only after they compound. The four audit dimensions (provenance coverage, attestation freshness, retrieval boundary compliance, key fact drift) form the structural complement to per-output monitoring; together they catch what neither alone can catch. The Quarterly Memory Audit Worksheet turns the framework into a repeatable practice that takes one engineer-week per quarter and prevents incidents whose response cost would dwarf the audit by orders of magnitude. Set the cadence, name the owner, run the worksheet, surface the findings, track the remediations. The audit is the cheap option; skipping it is the expensive one. Pick deliberately.