How to Test Generative UI (AI That Renders Its Own Interface)

Somebody on your team wrote getByTestId('price-chart') six weeks ago. It passed for a month. Then the model, asked the exact question a user had asked a hundred times before, decided a table communicated the answer better this time. The test didn't fail because the feature broke. It failed because the assumption underneath it, that a price-chart exists to be found, was never true. Nobody authored that ID. The model did, fresh, on this render only.

That's the whole problem in one sentence, and it's why "write better selectors" doesn't help. There's nothing to snapshot when several answers are correctly right.

What Generative UI Is (and Why Selectors Break)

Generative UI is an interface where the model returns structure, not just text: a component name, a set of props, sometimes a whole tree, and the app renders whatever came back. Ask a static app "how did revenue do this quarter" and it renders the one dashboard a developer built, every time. Ask a generative-UI app the same question and the model might return a bar chart this run, a table next run, and a short paragraph the run after, because all three are legitimate answers and nothing commits to one shape ahead of time.

Every conventional UI-testing method, test IDs, selectors, visual-regression snapshots, rests on one assumption: a human authored the markup, so it's stable enough to pin a selector to or diff against a golden image. Generative UI removes the human from that step. There's no golden image, because there's no single correct image. A snapshot test here doesn't catch regressions. It fails on entirely correct behavior, which trains the team to ignore red builds.

Testing the Generation Contract: Component Correctness Before Render

The fix is deceptively simple: stop treating "did the model produce a correct interface" as one question. Split it in two. Did the model choose a valid component with valid props for this intent, checked against a schema, no rendering involved? And separately, given that component and those props, does it render, is it interactive, is it accessible? The first is about the model's decision. The second is about your code. Conflating them is why teams debug a rendering bug when the model picked a chart with a string where a number belonged, or debug a "model" bug when a Chart component was simply broken and nobody had ever rendered it in a test.

Constrain the model to a fixed registry of allowed components, each with a schema, and validate every generation against it before a renderer ever sees it. Here's the registry, built with zod so the schema and the TypeScript types share one source:

import { z } from "zod";

const ChartSpec = z.object({
  component: z.literal("Chart"),
  props: z.object({
    title: z.string().min(1).max(80),
    kind: z.enum(["bar", "line", "pie"]),
    series: z
      .array(
        z.object({
          label: z.string().min(1).max(40),
          value: z.number()
        })
      )
      .min(1)
      .max(24)
  })
});

const TableSpec = z.object({
  component: z.literal("Table"),
  props: z.object({
    columns: z.array(z.string().min(1).max(40)).min(1).max(8),
    rows: z.array(z.array(z.union([z.string(), z.number()]))).min(1).max(200)
  })
});

const CardListSpec = z.object({
  component: z.literal("CardList"),
  props: z.object({
    cards: z
      .array(
        z.object({
          heading: z.string().min(1).max(60),
          body: z.string().min(1).max(280),
          actionLabel: z.string().min(1).max(24).optional()
        })
      )
      .min(1)
      .max(12)
  })
});

const ProseSpec = z.object({
  component: z.literal("Prose"),
  props: z.object({
    text: z.string().min(1).max(1200)
  })
});

export const GeneratedUISpec = z.discriminatedUnion("component", [
  ChartSpec,
  TableSpec,
  CardListSpec,
  ProseSpec
]);

export type GeneratedUISpec = z.infer<typeof GeneratedUISpec>;

export function validateGeneratedUI(raw: unknown) {
  return GeneratedUISpec.safeParse(raw);
}

With the registry in place, the generation contract is a deterministic test with no rendering at all: call the generator for a fixed intent repeatedly, and assert every result parses against the schema.

import { describe, it, expect } from "vitest";
import { validateGeneratedUI } from "../../src/registry/componentRegistry";
import { generateUIForIntent } from "../../src/generation/generateUI";

const SAMPLE_COUNT = 20;
const INTENT = "show quarterly revenue by region";

describe("generation contract: quarterly revenue intent", () => {
  it(`produces a registry-valid component on ${SAMPLE_COUNT} independent samples`, async () => {
    const failures: string[] = [];

    for (let i = 0; i < SAMPLE_COUNT; i++) {
      const raw = await generateUIForIntent(INTENT, { seed: i });
      const result = validateGeneratedUI(raw);

      if (!result.success) {
        const issues = result.error.issues.map((issue) => issue.message).join("; ");
        failures.push(`sample ${i}: ${issues}`);
      }
    }

    expect(failures, `invalid generations:\n${failures.join("\n")}`).toHaveLength(0);
  });
});

A failure here means one thing: the model chose an invalid shape, a component outside the registry, a wrong-typed prop, a list past its length bound. It is not a rendering bug and not a flake. It's a prompt or registry problem, and the message should point straight at it.

Layout-Change Resilience: Assert on Semantics, Not Structure

A registry-valid generation still has to render correctly, and this is where the assertion style matters most. Query by accessible role and name, read the data the user can see, and never touch DOM depth, sibling order, or a CSS class. The registry already guarantees the component is one of a known, finite set. Your render test's job is proving that whichever one shipped, a user, or a screen reader, can find it, read it, and act on it.

Here's that render test against a CardList fixture: it asserts a heading exists by name, the body text is present, every action button is reachable by keyboard, and it never references a class name or a DOM position.

import { describe, it, expect } from "vitest";
import { render, screen } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import { GeneratedComponent } from "../../src/components/GeneratedComponent";
import type { GeneratedUISpec } from "../../src/registry/componentRegistry";

const cardListFixture: GeneratedUISpec = {
  component: "CardList",
  props: {
    cards: [
      {
        heading: "West region",
        body: "Revenue up 12% quarter over quarter.",
        actionLabel: "View details"
      },
      {
        heading: "East region",
        body: "Revenue flat, down 1% quarter over quarter.",
        actionLabel: "View details"
      }
    ]
  }
};

describe("GeneratedComponent: CardList render", () => {
  it("exposes each card as an accessible region with a reachable action", async () => {
    const user = userEvent.setup();
    render(<GeneratedComponent spec={cardListFixture} />);

    expect(screen.getByRole("heading", { name: /west region/i })).toBeInTheDocument();
    expect(screen.getByText(/revenue up 12%/i)).toBeInTheDocument();

    const actions = screen.getAllByRole("button", { name: /view details/i });
    expect(actions).toHaveLength(2);

    await user.tab();
    expect(actions[0]).toHaveFocus();
    await user.keyboard("{Enter}");
  });
});

Notice what's brittle here and what isn't. screen.getByRole("heading", { name: /west region/i }) survives a redesign that changes the heading's font size, wrapper element, or position. It only breaks if the accessible name changes, which is the one thing you want a test to catch.

Accessibility of Generated Markup: The Regression Nobody Reviews

This is the layer with the least coverage anywhere. Every other line of markup in your app went through a human: someone wrote it, ideally someone reviewed it for accessibility. Generated markup skips both steps. The model authors fresh DOM per request, nobody reads the diff, and no PR catches a missing label or an unreachable button. If one prop combination produces a card with an icon-only action and no accessible name, that ships the first time a user's intent triggers it, and keeps shipping until someone runs an automated check.

The fix mirrors the generation contract: sweep every variant the registry allows with an automated tool on every run. axe-core is the standard, and running it isn't extra credit, it's the only mechanism that catches an a11y regression in markup nobody wrote and nobody reviewed.

import { describe, it, expect } from "vitest";
import { render } from "@testing-library/react";
import { axe } from "vitest-axe";
import { GeneratedComponent } from "../../src/components/GeneratedComponent";
import type { GeneratedUISpec } from "../../src/registry/componentRegistry";

const fixtures: { name: string; spec: GeneratedUISpec }[] = [
  {
    name: "chart",
    spec: {
      component: "Chart",
      props: {
        title: "Revenue by quarter",
        kind: "bar",
        series: [
          { label: "Q1", value: 120000 },
          { label: "Q2", value: 134000 }
        ]
      }
    }
  },
  {
    name: "table",
    spec: {
      component: "Table",
      props: {
        columns: ["Region", "Revenue"],
        rows: [
          ["West", 134000],
          ["East", 98000]
        ]
      }
    }
  },
  {
    name: "card list",
    spec: {
      component: "CardList",
      props: {
        cards: [
          { heading: "West region", body: "Revenue up 12%.", actionLabel: "View details" }
        ]
      }
    }
  },
  {
    name: "prose",
    spec: { component: "Prose", props: { text: "Revenue rose 12% quarter over quarter." } }
  }
];

describe("a11y sweep: every registry-allowed component variant", () => {
  for (const fixture of fixtures) {
    it(`has no axe violations for ${fixture.name}`, async () => {
      const { container } = render(<GeneratedComponent spec={fixture.spec} />);
      const results = await axe(container);
      expect(results.violations, JSON.stringify(results.violations, null, 2)).toHaveLength(0);
    });
  }
});

Pair the axe sweep with an explicit keyboard-reachability assertion for every interactive element the registry allows, the way the render test above tabs to a button and activates it with Enter. Axe catches missing labels, contrast failures, and invalid ARIA. It does not catch "this button exists, has a label, and is still unreachable because nothing in the tab order lands on it." You need both.

How to Test Generative UI End to End: Behavioral E2E on a Non-Static DOM

Everything above tests a component in isolation. The last layer drives the real, running application: send the intent, wait for whatever rendered this time, and assert on the user-visible outcome and the side effect, never on shape. This is also where non-determinism becomes a testing discipline, not a rendering concern. Run the same intent repeatedly and assert on the invariant, every variant is a valid component, every variant is accessible, every variant conveys the correct answer, not on one blessed render seen once in a demo.

Here's the behavioral E2E spec, run across ten independent samples of the same intent against a real deployed instance of the app, asserting the correct figure is legible in whatever container shipped and that the app's own audit log recorded the response:

import { test, expect, type Page } from "@playwright/test";

const INTENT = "show quarterly revenue by region";
const SAMPLE_COUNT = 10;

async function submitIntent(page: Page, intent: string) {
  await page.getByRole("textbox", { name: /ask a question/i }).fill(intent);
  await page.getByRole("button", { name: /send/i }).click();
}

test.describe("generative UI behavioral E2E: quarterly revenue intent", () => {
  for (let i = 0; i < SAMPLE_COUNT; i++) {
    test(`sample ${i}: renders a valid, correct answer`, async ({ page }) => {
      await page.goto("/");
      await submitIntent(page, INTENT);

      const result = page.getByRole("region", { name: /assistant response/i });
      await expect(result).toBeVisible({ timeout: 15_000 });

      // Semantic outcome: whichever layout shipped, the correct figure is legible somewhere in it.
      await expect(result).toContainText(/\$[\d,]+/);

      // Side effect: the app's audit log recorded this response, independent of which shape rendered.
      const auditLog = page.getByRole("log", { name: /audit log/i });
      await expect(auditLog.getByText(/quarterly revenue by region/i)).toBeVisible();
    });
  }
});

How Autonoma Verifies What the Model Actually Rendered

A schema check proves the model picked a valid shape. A render test proves that shape is interactive and accessible. Neither proves the feature works in the running app, because the layer that matters most, does this answer actually appear correctly, wherever the user is looking, changes shape every render by design. A hand-scripted test built around one screenshot can't survive that.

This is the layer Autonoma runs against the real, deployed app. Our Planner reads the codebase and the intents the UI exposes, then plans end-to-end cases from that, including the database state each case needs, rather than pinning a case to one remembered render. The Executor drives the live app the way a user would, across repeated runs of the same intent. The Reviewer classifies each result: a genuine bug, an agent error, or a case that no longer matches the plan, which matters here since a differently shaped but valid render should never be misclassified as a failure. As the registry evolves, the Diffs Agent updates the suite from each pull request's diff.

Gating Generative UI in CI

None of the four layers earn their keep until they're wired into a build that blocks a merge. Run the generation contract first, since it's the fastest and narrows failures to a prompt or schema problem before anything else spends time on rendering. Run the render and accessibility suites next, since they're still deterministic and don't need a deployed environment. Run the behavioral suite last, against a real preview or staging deployment, sampling each intent enough times to trust the result.

name: generative-ui-tests

on:
  pull_request:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    env:
      SAMPLE_COUNT: 20
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install dependencies
        run: npm ci

      - name: Generation contract (schema validation across N samples)
        run: npm run test:contract

      - name: Render and accessibility (every registry-allowed variant)
        run: npm run test:render && npm run test:a11y

      - name: Install Playwright browsers
        run: npx playwright install --with-deps chromium

      - name: Build the app
        run: npm run build

      - name: Start the app
        run: npm run start &

      - name: Wait for the app to be ready
        run: npx wait-on -t 60s http://localhost:3000

      - name: Behavioral E2E (N samples of the same intent, live app)
        run: npm run test:e2e

Set the sample count deliberately. Ten to twenty samples per intent is enough to catch a registry gap or a genuinely broken component without turning the pipeline into a marathon; raise it for intents with more branching in the registry, and lower it once a suite has run clean for weeks. The number that matters isn't how many times you sampled. It's whether every sample, no matter which valid shape it took, told the user the truth.

When this gate goes flaky, resist the reflex to add a retry. A flaky generative-UI suite means one of two things, and a retry hides which one it is: either the assertion is over-specified and quietly depends on a shape the registry never promised, or the registry itself is under-constrained and is letting through a component or prop combination nothing downstream was built to handle. Fix the assertion or fix the registry. Don't paper over either with a second attempt.

This cluster's guide to testing generative AI applications covers the broader discipline this fits into, and if your interface streams its answer in progressively rather than rendering all at once, testing streaming AI responses covers that layer specifically. The selector-fragility problem itself isn't new to generative UI, either, it's the same root cause behind self-healing test automation in ordinary web apps, just sharper here because the layout changes on purpose, every time. And once the interface itself is solid, testing an AI copilot is the natural next layer up, verifying the assistant embedded inside your product rather than the raw components it renders.

The Suite Has to Change as Fast as the Registry

Build the bottom two layers yourself. A Zod registry and an axe sweep are cheap, fast, deterministic, and they catch most of what breaks. The layer that resists being hand-built is the top one, and not because driving a browser is hard. It's because a generative UI invalidates hand-written end-to-end tests faster than any other kind of interface. Add one component to the registry and every behavioral assertion that implicitly assumed the old set of possible shapes is now under-specified, without failing, and without anyone noticing. The suite still runs green while covering strictly less than it did last week.

That decay is the specific problem Autonoma is built for, and it's why the behavioral layer is the one worth handing off. Tests run against the real deployed application in a preview environment on the pull request, so a valid-but-differently-shaped render is exercised as a live render rather than compared against a remembered one. The Diffs Agent updates the suite from each pull request's diff, which is what keeps a registry change and its coverage on the same schedule instead of drifting apart between releases.

There's a discipline point underneath all of it worth stating plainly. Every layer here works only because the assertions describe what a user must be able to do, not what the DOM must look like. Write them that way and a generative UI is genuinely testable, in CI, on every pull request. Write them the old way and you'll spend the next quarter chasing red builds caused by nothing being wrong at all. Autonoma is built on that same principle, which is exactly why it holds up against an interface that redraws itself: the assertion follows the user's intent, and whichever valid shape the model picked this run is free to change. Nobody has to author getByTestId('price-chart') ever again, which is the right outcome, because nobody authored the chart either.

Frequently Asked Questions