How to Test Multi-Turn Conversations and Context Retention

Turn fourteen was where it broke. On turn two, a user setting up a booking said, "I'm booking for my daughter Maya, she's 7." Turns three through eleven wandered into refund windows, baggage limits, and seat selection, none of it about Maya. On turn fourteen the user circled back: "ok so can she bring a friend?" The bot answered as if "she" referred to nobody in particular, and quoted the policy for an unaccompanied adult passenger.

Graded turn by turn, every reply in that transcript was defensible. The refund answer was correct. The baggage answer was correct. Turn fourteen's answer, read alone with no history attached, was even coherent. The failure only exists at the level of the conversation, and a test suite that scores replies one at a time will never see it.

That's the gap this piece fills. Response-level evals, the kind most teams already have, check whether a reply is good given the transcript up to that point. They say nothing about whether the transcript itself survived the trip. What follows are three specific ways a conversation degrades that a single-turn eval structurally cannot catch, plus a runnable pattern for testing each one.

It's worth naming the second gap up front, because it shapes where these tests stop. Everything below reads the conversation. None of it touches the product the conversation is driving, and a booking bot that holds Maya's name perfectly across fourteen turns can still leave the reservation unchanged. That distinction, between a coherent transcript and a correct application, is the one Autonoma was built around, and it comes back at the end of this piece with a concrete example.

Context Retention Is Not Memory, and Testing Them the Same Way Is a Mistake

Two very different things get called "the bot remembered." In-window context retention is whatever the model can still see inside the current conversation's token window. It lives entirely inside one session and dies the moment that session ends: test it by scripting turns inside a single conversation and asserting on what survives from one turn to the next. Persistent memory is whatever the system deliberately wrote to a durable store and retrieves later, in a separate session starting from zero context: test that by closing the first session, opening a second, and asserting the system pulls the right fact back out.

Conflating the two is a common mistake. A bot can pass every multi-turn test here and still have zero persistent memory, holding a fact across turn six of one conversation says nothing about whether it wrote anything down for next Tuesday. The reverse is just as real: a working long-term memory store doesn't stop a bot from losing the plot by turn six of a single session, since retrieval-from-storage and holding-state-in-context run through entirely different code paths.

This piece covers the first column only. For the store-and-retrieve-later layer, how to test AI agent memory is the sibling piece, and it's worth reading both if your bot claims to do either.

The Harness: A Scripted User, a System Under Test, and a Judge

Every pattern below reuses the same three-actor shape: a scripted user actor, the system under test, and a judge layer that asserts on the conversation rather than on any one reply's wording. If you haven't built that harness yet, agent simulation testing is the full derivation, three actors, a tool-failure scenario, and why a judge beats exact match. This piece extends that shape for multi-turn state specifically.

The seam worth naming up front is send(conversation, message): it appends a user turn, calls your chat client with the full history, appends the reply, and returns it. Every pattern below is written against that seam so it drops into whatever you're running, your own agent server, a hosted API, a wrapped SDK, without rewriting the tests. If you haven't covered single-turn correctness yet, how to test a chatbot is the fundamentals piece to start from; what follows is about what changes once turns start accumulating.

Here's the shared fixture layer every test file below builds on, the Conversation object, the send() seam, and a pinned_facts fixture for the facts a session establishes early and needs to keep:

class Conversation:
    """A running multi-turn conversation: a list of {role, content} messages."""

    def __init__(self):
        self.messages = []

    def user(self, text):
        self.messages.append({"role": "user", "content": text})
        return self

    def assistant(self, text):
        self.messages.append({"role": "assistant", "content": text})
        return self

    def history(self):
        return list(self.messages)


def _call_your_model(history):
    # Placeholder: point this at your actual chat completion endpoint.
    # It must accept the full message history (a list of {role, content}
    # dicts) and return a string reply. Wire this to your own client
    # (OpenAI, Anthropic, your own agent server, whatever you run in prod).
    raise NotImplementedError("Wire this to your own chat client.")


def send(conversation, message):
    """The seam. Swap _call_your_model for your own client call.

    Appends `message` as a user turn, sends the full conversation history
    to your system under test, appends the reply as an assistant turn,
    and returns the reply text.
    """
    conversation.user(message)
    reply = _call_your_model(conversation.history())
    conversation.assistant(reply)
    return reply


def run_scripted_turns(conversation, turns):
    """Run a list of user messages through `send`, returning all replies in order."""
    replies = []
    for turn in turns:
        replies.append(send(conversation, turn))
    return replies


def assert_pass_rate(check_fn, n_runs, threshold, label="check"):
    """Run `check_fn(i)` n_runs times and assert the pass rate clears threshold.

    `check_fn` takes the run index and returns True (pass) or False (fail).
    Multi-turn tests are inherently probabilistic and failure compounds per
    turn, so a single 1/1 run is the wrong bar. This asserts on the rate
    across N runs instead, with the failure count and threshold folded into
    the message so a red result is actually diagnosable.
    """
    failures = []
    for i in range(n_runs):
        if not check_fn(i):
            failures.append(i)

    passes = n_runs - len(failures)
    pass_rate = passes / n_runs
    assert pass_rate >= threshold, (
        f"{label}: {passes}/{n_runs} runs passed ({pass_rate:.0%}), "
        f"required >= {threshold:.0%}. Failing runs: {failures}"
    )

The fixtures themselves stay thin, two of them, wrapping the harness above so every test file gets a fresh conversation and the facts it pins at the start:

import pytest

from lib.harness import Conversation


@pytest.fixture
def conversation():
    return Conversation()


@pytest.fixture
def pinned_facts():
    """Facts a conversation establishes early that must survive the whole session."""
    return {
        "child_name": "Maya",
        "child_age": "7",
        "booking_type": "single passenger, one child",
    }

Failure Mode 1: Topic-Switch-Then-Return

This is the turn-fourteen story from the top of this piece, and it's the single most common real-world multi-turn break, precisely because almost nobody tests for it directly. The shape is always the same: establish a fact, spend several turns on something else entirely, then refer back with a pronoun instead of restating it. A human conversation partner does this constantly and never notices. A bot with weak context handling drops the referent the moment the topic moves on.

The assertion is not on the words of the final reply. It's on whether the reply's meaning still resolves the pronoun to the right referent, which is exactly why this needs a judge instead of a string match: "she can bring a friend if the friend is also a listed passenger" and "yes, one additional passenger is allowed on Maya's booking" are both correct and share almost no words.

Here's the test: establish the fact, insert three unrelated turns, then probe the referent, and assert the judge sees it survive:

from lib.harness import Conversation, run_scripted_turns, assert_pass_rate
from lib.judge import judge_coreference_survived

SCRIPTED_TURNS = [
    "I'm booking a seat for my daughter Maya, she's 7 years old.",
    "What documents does she need to fly alone?",
    "Actually, before that, what's your refund policy if we need to cancel?",
    "And is there a change fee if we move the date instead?",
    "One more thing, do you charge extra for checked bags?",
    "Ok, so can she bring a friend on the same booking?",
]

ASSERTION_TURN_INDEX = 5  # "can she bring a friend?" is turn 6, 0-indexed here

N_RUNS = 5
REQUIRED_PASS_RATE = 0.8  # four of five runs must preserve coreference


def test_topic_switch_then_return_preserves_coreference(pinned_facts):
    def run_once(_i):
        conversation = Conversation()
        replies = run_scripted_turns(conversation, SCRIPTED_TURNS)
        final_reply = replies[ASSERTION_TURN_INDEX]

        # The assertion is on coreference survival, not on the words of the
        # reply. "she" at turn 6 must still resolve to the child established
        # at turn 1, even though turns 3-5 were about a different topic.
        return judge_coreference_survived(
            conversation_history=conversation.history(),
            pronoun_turn=SCRIPTED_TURNS[ASSERTION_TURN_INDEX],
            expected_referent=pinned_facts["child_name"],
            reply=final_reply,
        )

    assert_pass_rate(
        run_once,
        N_RUNS,
        REQUIRED_PASS_RATE,
        label="topic-switch-then-return coreference",
    )

Failure Mode 2: Context-Window Overflow

Every model, and every wrapper around one, has a token budget, and every long enough conversation eventually crosses it. The wrong question is whether the bot breaks once that happens. Something always changes past the window, older turns get dropped, summarized, or both. The right question is whether it degrades gracefully. Acknowledging the gap, re-asking, or falling back to a summary that preserved the facts you pinned is an acceptable failure. Silently forgetting and then confidently inventing a replacement is the dangerous one, since it looks identical to a correct answer until someone checks it against reality.

Most teams discover their truncation strategy the hard way: it's "drop the oldest messages first," which quietly drops the most important turn in the conversation, the one where the user stated what they actually wanted. The fix most mature systems converge on is a pinned-facts layer, a short summary of the handful of facts that must never drop, refreshed and re-injected on every turn regardless of what else gets truncated.

Test the boundary on purpose. Pad a conversation with filler turns past your system's actual truncation threshold, then probe for a fact you pinned at the start, and assert one of two allowed outcomes, remembered, or acknowledged as gone, never a confidently invented substitute.

import pytest

from lib.harness import Conversation, run_scripted_turns, assert_pass_rate
from lib.judge import judge_contains_fact, judge_is_confident_claim

# Replace with your own system's actual context window / summarization
# truncation budget, measured in turns for this simplified harness.
TRUNCATION_THRESHOLD_TURNS = 30

PINNED_FACT_TURN = (
    "I'm flying under passenger ID 8823-A, please keep that on file "
    "for this whole conversation."
)
FILLER_TOPIC = "Tell me something interesting about airline loyalty programs in general."
PROBE_TURN = "What passenger ID am I flying under again?"

N_RUNS = 5
REQUIRED_PASS_RATE = 0.8


def build_overflow_script(pad_turns):
    script = [PINNED_FACT_TURN]
    script += [FILLER_TOPIC] * pad_turns
    script.append(PROBE_TURN)
    return script


@pytest.mark.parametrize(
    "pad_turns",
    [5, TRUNCATION_THRESHOLD_TURNS - 2, TRUNCATION_THRESHOLD_TURNS + 5],
)
def test_pinned_fact_survives_or_degrades_gracefully(pad_turns):
    script = build_overflow_script(pad_turns)

    def run_once(_i):
        conversation = Conversation()
        replies = run_scripted_turns(conversation, script)
        final_reply = replies[-1]

        remembered = judge_contains_fact(final_reply, "8823-A")
        if remembered:
            return True  # under the threshold, or summarization preserved the pin: fine

        # Past the threshold, forgetting is allowed. Confidently inventing a
        # different passenger ID is not. That's the silent-confabulation
        # failure.
        invented_a_fake_id = judge_is_confident_claim(final_reply) and not remembered
        return not invented_a_fake_id

    assert_pass_rate(
        run_once,
        N_RUNS,
        REQUIRED_PASS_RATE,
        label=f"context-window overflow (pad_turns={pad_turns})",
    )

Failure Mode 3: Long-Conversation Degradation

The first two failure modes happen at a specific turn. This one is a trend, and it needs a different assertion. Over twenty, forty, sixty turns, even comfortably inside the context window, quality decays: the bot drifts off its persona, repeats phrasing, stops following an instruction given early ("stop using bullet points" quietly stops holding by turn thirty), or contradicts a commitment it made turns ago.

The version of this that costs real money is the one where the drift is in the actions, not the prose. A bot forty turns into a support session that quietly stops applying a constraint the user set at turn three keeps writing perfectly reasonable replies while issuing credits it shouldn't. A transcript-level judge scores those replies as fine, because as sentences they are fine. Catching it needs an assertion on the account, not the answer, which is the layer Autonoma drives: the same long conversation replayed against the running product, with the invariant checked on what the product ended up in rather than on what the bot said about it.

A single assertion at the end misses this entirely, since the failure is gradual, not a step function. Score every turn against the same invariant, track the trend, and assert the score doesn't regress past a threshold as the conversation gets longer, plus a separate check for direct self-contradiction against anything said earlier in the session:

from lib.harness import Conversation, run_scripted_turns, assert_pass_rate
from lib.judge import score_instruction_adherence, judge_contradicts_earlier_turn

LONG_SCRIPT = [
    "From now on, answer every question without using bullet points or numbered lists.",
] + [
    f"Question {i}: what's one tip for a smoother trip through security?"
    for i in range(1, 41)
]

# Late-window adherence must not drop below this fraction of early-window adherence.
ADHERENCE_DECAY_FLOOR = 0.8

N_RUNS = 5
REQUIRED_PASS_RATE = 0.8


def test_instruction_adherence_does_not_regress_over_long_conversation():
    def run_once(_i):
        conversation = Conversation()
        replies = run_scripted_turns(conversation, LONG_SCRIPT)

        scores = [
            score_instruction_adherence(reply, rule="no_bullet_points")
            for reply in replies[1:]
        ]
        early_avg = sum(scores[:10]) / len(scores[:10])
        late_avg = sum(scores[-10:]) / len(scores[-10:])
        return late_avg >= early_avg * ADHERENCE_DECAY_FLOOR

    assert_pass_rate(
        run_once,
        N_RUNS,
        REQUIRED_PASS_RATE,
        label="long-conversation instruction adherence",
    )


def test_no_self_contradiction_against_earlier_commitment():
    def run_once(_i):
        conversation = Conversation()
        replies = run_scripted_turns(conversation, LONG_SCRIPT)
        history = conversation.history()

        for i, reply in enumerate(replies[10:], start=10):
            if judge_contradicts_earlier_turn(reply, history[: i * 2]):
                return False
        return True

    assert_pass_rate(
        run_once,
        N_RUNS,
        REQUIRED_PASS_RATE,
        label="long-conversation self-contradiction check",
    )

Why Multi-Turn Tests Are the Flakiest Tests You'll Write

Failure probability compounds per turn. A test with a 98% pass rate on any single assertion, run six times across a scripted conversation, has roughly an 89% chance all six hold, and every failure mode above needs several chained assertions, not one. Multi-turn tests will flake more than anything else in your suite, and that's expected.

Handle it like any genuinely probabilistic system. Assert on invariants, never exact wording, since two correct replies can use completely different words. Run each test N times and require a pass rate, not a single one-out-of-one pass; five runs with four clearing the bar is a defensible floor. Prefer a judge or embedding similarity over string comparison, and treat temperature=0 as a start, not a guarantee: retrieval order, summarization, and tool results can still vary between identical-looking runs even at zero temperature.

The heuristic worth pinning above your desk: a flaky multi-turn test means one of three things. Your assertion is too strict for normal variance, your prompt or script is ambiguous enough to have more than one right answer, or your context strategy actually is non-deterministic and the flake is real signal, not noise to average away. Here's the judge helper every test above calls into, boolean and scored questions instead of text comparison:

"""Semantic assertion helpers for multi-turn tests.

Exact string matching is the wrong tool here: two correct replies can use
completely different words. Everything below asks an LLM judge a narrow,
boolean or scored question instead of comparing reply text directly.
Swap `_ask_judge` for your own judge client.
"""


def _ask_judge(prompt, temperature=0):
    # Point this at your judge model of choice. Pin the model version and
    # keep temperature at 0, but don't treat that as determinism: retrieval
    # order, summarization, and tool results can still vary between calls.
    raise NotImplementedError("Wire this to your own judge client.")


def judge_coreference_survived(conversation_history, pronoun_turn, expected_referent, reply):
    prompt = (
        f"Conversation so far: {conversation_history}\n"
        f"Latest user turn: {pronoun_turn}\n"
        f"Latest assistant reply: {reply}\n"
        f"Question: does the reply treat the pronoun in the latest user turn as "
        f"referring to '{expected_referent}'? Answer only 'yes' or 'no'."
    )
    return _ask_judge(prompt).strip().lower().startswith("yes")


def judge_contains_fact(reply, fact):
    prompt = (
        f"Reply: {reply}\n"
        f"Question: does this reply state the fact '{fact}'? Answer only 'yes' or 'no'."
    )
    return _ask_judge(prompt).strip().lower().startswith("yes")


def judge_is_confident_claim(reply):
    prompt = (
        f"Reply: {reply}\n"
        "Question: does this reply state a specific factual claim with confidence, "
        "rather than hedging, acknowledging uncertainty, or asking a clarifying "
        "question? Answer only 'yes' or 'no'."
    )
    return _ask_judge(prompt).strip().lower().startswith("yes")


def score_instruction_adherence(reply, rule):
    prompt = (
        f"Reply: {reply}\n"
        f"Rule: {rule}\n"
        "Score from 0.0 to 1.0 how well the reply follows the rule. "
        "Reply with only the number."
    )
    return float(_ask_judge(prompt).strip())


def judge_contradicts_earlier_turn(reply, prior_history):
    prompt = (
        f"Prior conversation: {prior_history}\n"
        f"Latest reply: {reply}\n"
        "Question: does the latest reply contradict a commitment or fact stated "
        "earlier in the conversation? Answer only 'yes' or 'no'."
    )
    return _ask_judge(prompt).strip().lower().startswith("yes")

Running This in CI Without Drowning in Flaky Reruns

Not every one of these belongs on every pull request. The topic-switch and overflow tests are cheap, a handful of turns each, so they run fine per PR with a couple of automatic reruns for the compounding-flake problem above. The long-conversation suite is expensive, forty-plus turns times however many judge calls per turn, and belongs on a nightly schedule instead, where a slower, fuller run is affordable.

Budget for judge calls specifically: a forty-turn conversation scored on one invariant per turn is forty extra model calls beyond the conversation itself, and that adds up fast running nightly. Gate on pass rate at the suite level the same way you gate on it per test, and treat a regressed rate, not any single flake, as the signal worth blocking a merge over:

name: multi-turn-tests

on:
  pull_request:
  schedule:
    - cron: "0 6 * * *"  # nightly full run

jobs:
  fast-checks:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt
      - name: Run topic-switch and overflow tests (cheap, per PR)
        run: pytest tests/test_topic_switch_return.py tests/test_context_window_overflow.py --reruns 2

  nightly-full-suite:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt
      - name: Run full suite including long-conversation degradation
        run: pytest tests/ --reruns 2

The Practitioner Takeaway

Every test pattern above proves the same narrow thing: the conversation, as text, held together. That's real and worth proving, and it's also not the whole application. A booking bot can pass all three failure-mode tests here, hold Maya's name across the topic switch, degrade gracefully past the window, stay in persona for sixty turns, and still leave the booking showing two passengers after the reply said "okay, removed." The transcript was coherent. The app state wasn't, and no test that only reads the conversation catches that gap.

We built Autonoma for exactly that layer. It runs behavioral end-to-end tests against the running application in a preview environment on the pull request, driving the conversation the way a user would and then asserting on what the product actually holds: the passenger count on the booking, the credit on the account, the ticket's owner. Every claim a reply makes about the world becomes checkable against the world. The bot saying "okay, removed" stops being evidence of anything, which is the correct amount of weight to give it.

The maintenance side matters as much as the assertion for a conversational feature, because prompts and flows change weekly. Autonoma's Diffs Agent reads each pull request's diff and updates the suite alongside the code, so a scripted conversation that no longer matches the product gets revised rather than quietly asserting last month's flow. A stale multi-turn suite is a particularly expensive kind of stale, since each scenario is long, hand-built, and easy to trust on reputation alone.

So script the three failure modes above into your suite and gate on pass rate instead of a single run. That catches the class of bug turn-by-turn evals were never built to see. Then put Autonoma underneath it, because the conversation holding together and the application doing what it said are two separate claims, and only one of them has a user's money attached. Turn fourteen is worth catching. The booking that never changed is worth catching more.

Frequently Asked Questions