How to Test AI Agent Memory (Short- and Long-Term)

How to test AI memory starts with one definition: verify that an agent stores, retrieves, and correctly applies information from earlier in the same session and from entirely separate sessions, not just that it writes a plausible-sounding reply. That means testing four things directly: recall across distractor turns, persistence into a brand-new thread, retrieval accuracy against decoy memories, and whether a newer fact correctly overrides an older, conflicting one.

A support agent we watched during a review call greeted a returning user by name: "Hey Priya, welcome back." Two days later, same user, a brand-new conversation: "I don't have your name on file, who am I speaking with?" The test suite was green the whole time: every test opened a fresh conversation, asked one question, checked the reply, and never asked the agent to remember something and come back later.

That gap is invisible from inside a single conversation. A multi-turn test catches the agent losing track of something said three messages ago. It doesn't catch it losing track of something said three days ago in a different thread: a different mechanism failing, not the context window, the memory store behind it.

Memory is also where AI-native testing stops resembling model evaluation and starts resembling application testing. Nothing about the greeting above is a language problem. The reply was well-formed both times; a row was written or it wasn't, a query matched or it didn't. That's a stateful application bug wearing a chatbot's clothes, which is why the coverage that actually catches it looks like end-to-end testing rather than scoring. It's the reasoning behind how we built Autonoma, and it shapes every assertion in this piece.

Memory Is Not Context

Context retention is what survives inside the model's context window during one conversation: the last several turns, still visible when it replies. Memory is persistent storage that outlives the window and the session: written once, retrievable in a conversation that hasn't even started yet.

Conflating the two is exactly why memory bugs slip past context-focused test suites. A context-window failure is a truncation or eviction problem: something got pushed out of the prompt before the model needed it again. A memory failure is a write-or-retrieve problem: either the fact never made it into storage, or it did, and the retrieval query never matched it.

A context bug loses something the model could technically still see. A memory bug loses something the model was never given a chance to see again.

Recall inside one conversation and recall across two separate ones are not the same test. This piece covers the second: within-session recall past distractors, cross-session persistence, retrieval accuracy against near-miss memories, and staleness. For the context-window layer itself, truncation and how far a long conversation degrades before the model loses the thread, see how to test multi-turn conversations.

Testing Within-Session Recall

Within-session recall tests AI memory retention inside one conversation: does the agent still have a fact once it's no longer the topic on screen? State a fact, inject a handful of unrelated turns, then ask something answerable only using the fact from several turns back.

If you already run a multi-turn harness (a user simulator, an agent, a judge scoring the transcript), memory assertions bolt onto the same loop. See agent simulation testing for that harness in raw Python.

Don't assert on exact wording: "the deploy window is Tuesdays 2 to 4pm" and "you're set for Tuesday afternoon, 2 to 4" are both correct, and an exact-match assertion will flake between them for reasons that have nothing to do with memory. Assert on the fact's presence semantically instead.

Here's the memory client and agent interface every test in this piece shares: a persistent store, a fresh-namespace helper for test isolation, and an agent that writes and retrieves against it.

"""Vendor-neutral memory store and agent interface shared by every test
in this repo.

`deterministic_llm` needs no API key, so the whole suite runs in CI with
zero external dependencies. Swap `llm_fn` on the `Agent` constructor for
a real provider call to exercise genuine model non-determinism; keep
every test's assertions exactly as they are.
"""

import re
import uuid
from dataclasses import dataclass
from typing import Callable, Optional


@dataclass
class MemoryRecord:
    key: str
    value: str


class MemoryStore:
    """A minimal persistent store, keyed by user id.

    Swap this for Redis, Postgres, or a vector DB. Every test in this
    repo only depends on write/read/read_all, so the storage backend
    underneath is irrelevant to the tests themselves.
    """

    def __init__(self):
        self._data: dict[str, list[MemoryRecord]] = {}

    def write(self, user_id: str, key: str, value: str) -> None:
        existing = self._data.setdefault(user_id, [])
        # A new fact under the same key overwrites the old one, so
        # staleness resolves correctly: the newest write always wins,
        # instead of both versions coexisting forever.
        self._data[user_id] = [r for r in existing if r.key != key]
        self._data[user_id].append(MemoryRecord(key=key, value=value))

    def read_all(self, user_id: str) -> list[MemoryRecord]:
        return list(self._data.get(user_id, []))

    def read(self, user_id: str, key: str) -> Optional[str]:
        for record in reversed(self.read_all(user_id)):
            if record.key == key:
                return record.value
        return None

    _STOPWORDS = {
        "the", "a", "an", "is", "are", "was", "were", "to", "of", "and",
        "or", "for", "in", "on", "at", "do", "does", "i", "you", "your",
        "my", "our", "me", "s", "again", "about", "what", "when", "how",
    }

    def search(self, user_id: str, query: str, top_k: int = 1) -> list[MemoryRecord]:
        """A stand-in for embedding-based retrieval: score every stored
        record by word overlap with the query (stopwords excluded), and
        return the best matches. Real vector search fails the same way
        this does when a query is ambiguous between two similar
        memories, which is exactly the failure mode the
        retrieval-accuracy tests target.
        """
        query_words = set(re.findall(r"\w+", query.lower())) - self._STOPWORDS
        scored = []
        for record in self.read_all(user_id):
            record_words = set(re.findall(r"\w+", f"{record.key} {record.value}".lower())) - self._STOPWORDS
            overlap = len(query_words & record_words)
            if overlap > 0:
                scored.append((overlap, record))
        scored.sort(key=lambda pair: pair[0], reverse=True)
        return [record for _, record in scored[:top_k]]


def new_namespace() -> str:
    """A fresh, unique user id for one test run.

    Cross-session tests deliberately share a user id between a "write"
    session and a "read" session. If two separate test runs share that
    same id, a leftover fact from a previous run can make a broken test
    pass for the wrong reason. Call this once per test, never once per
    suite, and never hardcode a user id in a cross-session test.
    """
    return f"test-user-{uuid.uuid4().hex[:12]}"


def deterministic_llm(system_prompt: str, message: str) -> str:
    """A zero-dependency stand-in for a real model call.

    It looks for memory values injected into the system prompt and
    echoes the relevant ones back in a plain sentence. A real agent
    phrases this differently every run, which is exactly the
    non-determinism the semantic assertions in this repo are built to
    handle instead of exact string matching.
    """
    facts = re.findall(r"\[MEMORY\] (.+)", system_prompt)
    if not facts:
        return "I don't have any information about that yet."
    return "Based on what you told me, " + "; ".join(f.lower() for f in facts) + "."


class Agent:
    """Wraps an LLM call and a MemoryStore behind one interface.

    `remember` is called explicitly, the same way a real agent's own
    memory-extraction step would write a fact after noticing it in a
    message. That keeps the tests honest about what they are exercising:
    retrieval and application of memory, not the extraction step itself.
    """

    def __init__(self, store: MemoryStore, llm_fn: Callable[[str, str], str] = deterministic_llm):
        self.store = store
        self.llm_fn = llm_fn

    def remember(self, user_id: str, key: str, value: str) -> None:
        self.store.write(user_id, key, value)

    def ask(self, user_id: str, message: str, top_k: int = 1) -> str:
        records = self.store.search(user_id, message, top_k=top_k)
        memory_block = "\n".join(f"[MEMORY] {r.value}" for r in records)
        system_prompt = f"You are a helpful assistant.\n{memory_block}"
        return self.llm_fn(system_prompt, message)

    def decide_action(self, user_id: str, request: str) -> dict:
        """A tiny behavioral decision, gated on a stored fact rather than
        on anything in the current message. This is what a context
        carryover test actually needs to check: not what the agent
        said, but what it did as a result of a fact from an earlier
        session.
        """
        plan = self.store.read(user_id, "plan_tier") or "unknown"
        if "enterprise" in request.lower() and plan != "enterprise":
            return {"action": "deny", "reason": f"requested action requires enterprise, user plan is {plan}"}
        return {"action": "allow", "reason": "no plan restriction matched"}

And the within-session recall test itself: state the fact, inject three distractor turns, then assert the reply carries the fact semantically.

from memory_client import Agent, MemoryStore, new_namespace


def test_recalls_a_fact_after_distractor_turns():
    """Within-session recall: state a fact, inject unrelated distractor
    turns, then ask a question that can only be answered correctly using
    the fact stated several turns earlier in the same conversation.
    """
    store = MemoryStore()
    user_id = new_namespace()
    agent = Agent(store)

    agent.remember(user_id, "deploy_window", "the team's deploy window is Tuesdays 2-4pm UTC")

    distractor_turns = [
        "What's the weather like for shipping hardware today?",
        "Can you summarize our last release notes?",
        "How do I reset my password?",
    ]
    for turn in distractor_turns:
        agent.ask(user_id, turn)

    reply = agent.ask(user_id, "When is our deploy window again?")

    # Assert on the fact's presence semantically, not on the model's
    # exact wording. "Tuesdays 2-4pm" and "Tuesday afternoon, 2 to 4"
    # are both correct; an exact-match assertion would flake between
    # them for reasons that have nothing to do with memory.
    assert "tuesday" in reply.lower()
    assert "2-4pm" in reply.lower()


def test_distractor_turns_alone_do_not_surface_the_fact():
    """A sanity check on the harness itself: a query with no overlap
    with any stored fact should not accidentally recall it.
    """
    store = MemoryStore()
    user_id = new_namespace()
    agent = Agent(store)

    agent.remember(user_id, "deploy_window", "the team's deploy window is Tuesdays 2-4pm UTC")

    reply = agent.ask(user_id, "How do I reset my password?")

    assert "tuesday" not in reply.lower()

Within-session recall reads from the same context window the fact was stated in. Cross-session memory has to survive that window being discarded entirely.

Testing Cross-Session Memory

Cross-session memory, the core of how to test long-term memory in AI agents, asks a different question: does the fact survive into a conversation that hasn't started yet, with a completely fresh context window? This is where most implementations fail silently. The write step succeeds, but the read-side retrieval query at the start of the next session never matches it, because it was phrased differently or scoped to the wrong session id instead of the wrong user id.

The test pattern: session A writes a fact. Tear the conversation down completely, no shared object, a genuinely new agent instance. Session B, same user id, reads. If the assertion only passes because session B reuses session A's in-memory state, the test is lying to you; the point is that nothing from session A survives except what made it into the persistent store.

The teardown is the assertion, which is worth stating plainly because it's also the thing hardest to fake at the harness level. A fresh Python object is a weaker teardown than a fresh process, and a fresh process is weaker than a fresh browser session against the deployed app with a real database behind it. Each step up closes a class of false pass. The strongest version of this test is the one Autonoma runs: drive the product through a real session, close it, come back in a new one, and check what the agent knows against what the application actually stored. At that level there is no shared state left to accidentally pass on, because the only thing connecting the two sessions is the user id and the database.

Here's that test, plus a case that looks like a memory bug and usually isn't: a query with no overlap returns the correct "I don't know" fallback, while the fact itself is still sitting untouched in the store.

from memory_client import Agent, MemoryStore, new_namespace


def test_fact_survives_a_brand_new_session():
    """Cross-session memory: session A writes a fact, the conversation
    is torn down completely, and a genuinely new session (a fresh Agent
    instance, no shared object) reads it back using the same user id.
    """
    store = MemoryStore()
    user_id = new_namespace()

    session_a = Agent(store)
    session_a.remember(user_id, "user_name", "the user's name is Priya")
    # session_a goes out of scope here. Nothing about it survives
    # except whatever made it into `store`.

    session_b = Agent(store)  # a fresh thread, fresh context window, same user id
    reply = session_b.ask(user_id, "Do you remember my name?").lower()

    assert "priya" in reply


def test_unrelated_query_in_a_new_session_does_not_falsely_report_amnesia():
    """The most common real-world bug looks like the opposite of this
    test: a write that succeeded, and a read-side query that never
    overlapped with it, so the agent looks like it "forgot" a fact that
    is actually still sitting in the store untouched.
    """
    store = MemoryStore()
    user_id = new_namespace()

    session_a = Agent(store)
    session_a.remember(user_id, "preferred_name", "call the user Pri")

    session_b = Agent(store)
    reply = session_b.ask(user_id, "What's today's date?").lower()

    # A question genuinely unrelated to the stored fact should return
    # the fallback. That's correct behavior, not a memory bug.
    assert "don't have any information" in reply

    # Confirm the fact itself is still there, untouched, so a future
    # query that actually matches it will still find it.
    assert store.read(user_id, "preferred_name") == "call the user Pri"

Retrieval Accuracy: The Decoy Problem

Persistence isn't the hard part. Getting the right memory back is. An agent with three stored addresses needs to retrieve the one matching the question, not a plausible neighbor or a hallucinated one. This dimension gets the least attention, because it only shows up once the store holds more than one similar fact.

Seed similar-but-distinct memories, query for one, and assert two things at once: the correct value is present, and the decoy values are absent. Most teams write the first assertion and skip the second, and the second is what actually catches a retrieval bug: a reply with the right address that also lists the wrong ones isn't passing, it's an agent that dumped its whole memory into the prompt and got lucky.

The same pattern covers the no-memory case: ask about something never stored, and the agent should say it doesn't know, not invent a plausible answer from whatever else is in context.

from memory_client import Agent, MemoryStore, new_namespace


def _seed_similar_addresses(agent, user_id):
    agent.remember(user_id, "address_nyc_project", "the NYC project address is 123 Lime Street, New York")
    agent.remember(user_id, "address_sf_project", "the SF project address is 500 Market Street, San Francisco")
    agent.remember(user_id, "address_home", "the user's home address is 9 Maple Ave, Austin")


def test_retrieves_the_correct_memory_and_not_the_decoys():
    """Retrieval accuracy is the dimension most memory tests skip. It is
    not enough to check that the right value showed up; the wrong,
    plausible neighbors have to be absent too, or the agent is dumping
    its whole memory into every reply and getting lucky.
    """
    store = MemoryStore()
    user_id = new_namespace()
    agent = Agent(store)
    _seed_similar_addresses(agent, user_id)

    reply = agent.ask(user_id, "What's the address for the NYC project?").lower()

    # Positive assertion: the correct value is present.
    assert "123 lime street" in reply

    # Negative assertion: the decoys are absent. This is the half of the
    # test everyone forgets to write.
    assert "500 market street" not in reply
    assert "9 maple ave" not in reply


def test_no_memory_exists_returns_a_fallback_not_a_guess():
    """When nothing in the store matches the question, the agent should
    say it doesn't know, not stitch together a confident-sounding
    answer from whatever else happens to be in memory.
    """
    store = MemoryStore()
    user_id = new_namespace()
    agent = Agent(store)
    _seed_similar_addresses(agent, user_id)

    reply = agent.ask(user_id, "What's my flight confirmation number?").lower()

    assert "don't have any information" in reply
    assert "123 lime street" not in reply
    assert "500 market street" not in reply
    assert "9 maple ave" not in reply


def test_updated_fact_overwrites_the_stale_one():
    """Staleness: when a newer fact conflicts with an older one, the
    newer one has to win outright, not coexist alongside it.
    """
    store = MemoryStore()
    user_id = new_namespace()
    agent = Agent(store)

    agent.remember(user_id, "city", "the user lives in Lisbon")
    agent.remember(user_id, "city", "the user moved to Berlin")

    reply = agent.ask(user_id, "What city do I live in?").lower()

    assert "berlin" in reply
    assert "lisbon" not in reply
    # One current fact per key, not a growing, ambiguous list.
    assert len(store.read_all(user_id)) == 1

The positive assertion is the easy half. The negative assertion, that the decoys never leak into the reply, is the one most memory tests skip.

Context Carryover: When Memory Changes Behavior

The hardest dimension isn't a string check at all. A user says "I'm on the Free plan" in one session; the test that matters isn't whether a later reply contains the word "free," it's whether that fact changes what the agent does: does it still offer an enterprise-only action to a user who said, sessions ago, they're not on that plan?

That's a behavioral assertion, not a text-match, and it needs an LLM-as-judge or an explicit rubric rather than a substring check, because the phrasing of a denial varies every run while the underlying decision doesn't. Write the criteria as something checkable: does the response withhold the action, does the reason reference the stored plan tier.

A green suite hides the most expensive bug right here: the reply reads fine while the underlying action is wrong, a Free-plan user routed into an enterprise-only workflow because the check only looked at words, never the state change behind them. That's the same gap Autonoma exists to close on the application side: our agents run behavioral end-to-end tests against the real running app, so a wrong action shows up as a wrong outcome in the product, not just a sentence that reads fine.

from memory_client import Agent, MemoryStore, new_namespace


def llm_judge(response: str, must_contain=None, must_not_contain=None) -> bool:
    """A stand-in for a real LLM-as-judge call.

    In production this is a separate model call graded against an
    explicit, checkable rubric, not a vague "was this appropriate"
    prompt. Swap the body for a real judge call; keep the rubric (the
    must_contain / must_not_contain lists) explicit either way.
    """
    text = response.lower()
    for phrase in must_contain or []:
        if phrase.lower() not in text:
            return False
    for phrase in must_not_contain or []:
        if phrase.lower() in text:
            return False
    return True


def test_free_plan_user_is_denied_an_enterprise_only_action():
    """Context carryover is a behavioral assertion, not a text-match. A
    fact stated in an earlier session has to change what the agent
    *does* in a later one, not just what it says. A reply can read
    perfectly fine while the underlying action is still wrong.
    """
    store = MemoryStore()
    user_id = new_namespace()

    session_a = Agent(store)
    session_a.remember(user_id, "plan_tier", "free")

    session_b = Agent(store)  # a brand-new thread; no shared context window
    decision = session_b.decide_action(user_id, "Can you generate the enterprise usage report?")

    assert decision["action"] == "deny"
    assert llm_judge(
        decision["reason"],
        must_contain=["free"],
        must_not_contain=["enterprise plan"],
    )


def test_enterprise_plan_user_is_allowed_the_same_action():
    """The negative case matters as much as the positive one: the same
    request should be allowed once the stored fact actually says
    enterprise, or the test is just checking "always deny" and calling
    it a memory test.
    """
    store = MemoryStore()
    user_id = new_namespace()

    session_a = Agent(store)
    session_a.remember(user_id, "plan_tier", "enterprise")

    session_b = Agent(store)
    decision = session_b.decide_action(user_id, "Can you generate the enterprise usage report?")

    assert decision["action"] == "allow"

Staleness: The Newer Fact Has to Win

A memory system that stores every version of a fact without resolving conflicts produces an agent that's technically correct and practically useless. The user said "I live in Lisbon" in March, "I moved to Berlin" in June. Query in July, and the answer has to be Berlin, not both, and not the older one because it got retrieved first.

Test this directly: write the same key twice with a conflicting value, assert the reply carries only the newer one. The retrieval-accuracy suite above already exercises this, since the store overwrites by key rather than appending, but it's worth asserting on by name: staleness bugs are usually a storage-layer decision made months before anyone writes a test for it.

Handling Non-Determinism in Memory Assertions

Every assertion here rides on a model call, and the same input produces different words each run. That's a different flavor of flaky than a brittle CSS selector, and loosening the assertion until it stops complaining is almost always the wrong fix.

Use semantic containment for anything the model is free to phrase. Reserve exact-match for hard values that must appear verbatim: an order id, a dollar figure. For the harder behavioral checks, use an LLM-as-judge graded against an explicit rubric. For anything genuinely variance-sensitive, run it five times and gate on a threshold, four out of five, rather than a single pass.

A flaky memory test almost always means one of two things: the assertion is stricter than the thing it's checking, or the memory prompt itself is ambiguous enough that the model has room to interpret it differently each run. Find out which before you "fix" it by loosening the pass bar.

Loosening a threshold hides a real bug behind a statistic. Tightening the persona or the judge's rubric fixes the actual ambiguity, and that's almost always where the fix belongs.

How Do You Wire Memory Tests Into CI?

Unit-level memory tests (within-session recall, retrieval accuracy, staleness) run against an in-memory store and belong in your normal CI job, on every pull request. Cross-session tests need a real persistence backend, since the point is confirming state survives a teardown, and an in-memory stub survives everything trivially, including bugs you'd want caught.

Give cross-session tests their own integration job against a disposable, per-run instance of that backend, and give every run a fresh, unique user id. Two runs sharing a user id pollute each other's memories: a test can pass because a previous run's leftover fact is still sitting there, worse than a red build.

Here's the gate: unit-level tests in the main job, cross-session tests split into a job that documents the real-backend requirement.

name: Memory Tests

on:
  pull_request:
  push:
    branches: [main]

jobs:
  unit:
    name: Within-session, retrieval-accuracy, and carryover tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pytest
      - run: >
          pytest test_within_session_recall.py test_retrieval_accuracy.py
          test_context_carryover.py -v

  cross_session_integration:
    name: Cross-session memory (needs a real persistence backend)
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pytest
      # This repo's MemoryStore is an in-process stub so the suite runs
      # with zero setup. In production this job points at your real
      # backend (Postgres, Redis, a vector store), because the point of
      # a cross-session test is confirming a fact survives a genuine
      # teardown, and an in-memory stub survives everything trivially,
      # including bugs you'd actually want it to catch.
      #
      # Every test calls new_namespace() for a fresh, unique user id.
      # Two runs sharing a user id pollute each other's memories and can
      # pass for the wrong reason, so this job should always run against
      # a clean, disposable instance of the backend, not a shared one.
      - run: pytest test_cross_session_memory.py -v

Memory vs. Context Retention, at a Glance

Dimension	Memory	Context Retention
What it is	Facts stored outside the model, across sessions	What survives inside one active context window
Where it lives	A database, cache, or vector store	The live prompt, this conversation only
How it fails	Write succeeds, retrieval query never matches	Older turns get truncated or evicted
How you test it	Cross-session read after a full teardown	Recall mid-conversation, before truncation
Which article covers it	This article	Multi-turn conversations testing

Testing whether an AI remembers previous conversations comes down to one discipline: write in one session, tear down, read in a genuinely new one, and assert semantically. Pair that with retrieval accuracy against decoys and a staleness check, and the memory layer gets the same coverage the rest of your application already has. For the multi-turn harness these tests can run inside of, see agent simulation testing; for the context-window layer itself, see how to test multi-turn conversations.

Where a Memory Suite Stops, and What Picks It Up

Every test above answers the same question: does the agent say the right thing given what it should remember. That's the correct question for a memory layer and the wrong question for the product wrapped around it. The Free-plan case from earlier is the tell. What matters isn't that the agent mentions the plan tier, it's whether the enterprise-only action stayed unavailable, whether the upgrade path rendered, whether the audit record shows what actually happened. Those live in the application, and a suite that reads replies can't reach any of them.

That's the layer Autonoma covers. It runs behavioral end-to-end tests against the real running application in a preview environment on the pull request, so a stored fact is verified by its effect on the product rather than by its appearance in a sentence. The two suites fit together cleanly: the pytest suite above pins the memory client's contract fast and offline on every commit, and the behavioral layer proves that contract still means something once the rest of the app is in the picture.

The maintenance argument is the one that usually decides it, though. Memory tests are unusually brittle for a reason that has nothing to do with models: they encode assumptions about storage keys, user scoping, and retrieval phrasing, and all three get refactored without anyone thinking of them as behavior changes. Autonoma's Diffs Agent updates the suite from each pull request's diff, which is what keeps a scoping change from turning a memory test into a test of nothing at all. That failure is quiet, and a quiet memory test is worse than none, because it reads as proof the Priya bug can't happen again.

Frequently Asked Questions

Test AI agent memory across four dimensions: within-session recall (does the agent still have a fact after several unrelated turns), cross-session persistence (does the fact survive into a brand-new thread with a fresh context window), retrieval accuracy (does the agent retrieve the right memory instead of a similar decoy or a hallucinated one), and staleness (does a newer, conflicting fact correctly override an older one). Assert on the fact's presence semantically rather than exact wording, since the model phrases the same correct answer differently every run.

Context is what survives inside the model's context window during one active conversation: recent turns still visible to the model when it replies. Memory is persistent storage that outlives both the window and the session: written once and retrievable in a conversation that hasn't started yet. A context failure is a truncation or eviction problem. A memory failure is a write-or-retrieve problem, and the two need different tests.

Write a fact in one session using a fresh, isolated user id, then tear that session down completely and start a genuinely new session, a new agent instance with no shared in-memory state, using the same user id. Ask a question in the new session that can only be answered correctly using the earlier fact. If the reply carries that fact semantically, cross-session memory is working. If it doesn't, check whether the write actually reached persistent storage before assuming the retrieval query is at fault.

AI session persistence testing verifies that information a user shared in one conversation is available and correctly applied in a later, unrelated session, not just recalled within the same conversation. It requires a genuine teardown between sessions, no shared context window or in-memory object, and a shared, stable user identifier so the read-side session can find what the write-side session stored.

An intermittent memory test failure usually means one of two things: the assertion is stricter than what it's actually checking, such as an exact-match assertion against a model's freely-phrased reply, or the underlying prompt or persona is ambiguous enough that the model has room to answer differently each run. Loosening the pass threshold hides the problem rather than fixing it. Tighten the wording of the stored fact, the query, or the judge's rubric first, and only add run-N-times gating for genuinely borderline behavioral checks.

Long-term memory testing is cross-session testing plus a time dimension. Write a fact in one session, tear the session down completely, and read it back in a genuinely new session using the same stable user id. Then add two assertions the short-term case does not need: that a newer conflicting fact overrides the older one rather than both being returned, and that the correct memory is still retrievable once the store holds many similar facts, not just the one you seeded. Run these against a real persistence backend, since an in-memory stub survives a teardown trivially and will pass even when the write never reached storage.

The suite is checking what the agent said rather than what the product did. A test can confirm the agent correctly recalls that someone is on the Free plan and still miss that the enterprise-only action stayed available, that the upgrade path never rendered, or that the audit record shows something different from the reply. Those outcomes live in the application, not in the sentence, so the assertion has to land there too. It's also why the teardown in a cross-session test matters: a fresh Python object is a weaker teardown than a fresh session against the deployed app with a real database behind it. Autonoma runs behavioral end-to-end tests against that real running application, so a stored fact gets verified by its effect on the product rather than by its appearance in a reply.

How to Test AI Agent Memory (Short- and Long-Term)

Memory Is Not Context

Testing Within-Session Recall

Testing Cross-Session Memory

Retrieval Accuracy: The Decoy Problem

Context Carryover: When Memory Changes Behavior

Staleness: The Newer Fact Has to Win

Handling Non-Determinism in Memory Assertions

How Do You Wire Memory Tests Into CI?

Memory vs. Context Retention, at a Glance

Where a Memory Suite Stops, and What Picks It Up

Frequently Asked Questions

How do you test AI agent memory?

What's the difference between memory and context in an AI agent?

How do I test if an AI remembers previous conversations?

What is AI session persistence testing?

Why does a memory test fail intermittently?

How do you test long-term memory in an AI agent?

My memory tests pass, but a user still got an action they shouldn't have. What's the gap?

How to Test AI Agent Memory (Short- and Long-Term)

Memory Is Not Context

Testing Within-Session Recall

Testing Cross-Session Memory

Retrieval Accuracy: The Decoy Problem

Context Carryover: When Memory Changes Behavior

Staleness: The Newer Fact Has to Win

Handling Non-Determinism in Memory Assertions

How Do You Wire Memory Tests Into CI?

Memory vs. Context Retention, at a Glance

Where a Memory Suite Stops, and What Picks It Up

Frequently Asked Questions

How do you test AI agent memory?

What's the difference between memory and context in an AI agent?

How do I test if an AI remembers previous conversations?

What is AI session persistence testing?

Why does a memory test fail intermittently?

How do you test long-term memory in an AI agent?

My memory tests pass, but a user still got an action they shouldn't have. What's the gap?

Related articles

How to Test an AI Agent (End to End)

CrewAI Evaluation and Testing: How to Test CrewAI Agents

What Is Agent Simulation Testing? A 3-Actor Harness

How to Test AI Agents That Take Actions (Tool Calls)