Pairwise testing (also called all-pairs testing) generates a small set of test cases that covers every possible pair of parameter values at least once, instead of every possible combination. On a 6-parameter, 4-value-each configuration matrix, the full cross product is 4,096 cases; a verified pairwise set covers the same 240 required pairs in 22. The technique doesn't decide what the correct output is for any row. It only changes how many rows you need to check.
If a job posting or interview loop mentioned pairwise testing and you want the real definition instead of a five-minute recap video, or you're onboarding a QA hire who needs a worked example instead of a slide deck, or an auditor asked how your team justifies not testing every configuration, this article is for you: how the technique tames a combinatorial explosion of configurations, a matrix you can check yourself, and a rule for when the reduction is worth adopting.
It's not for the QA lead deciding how much test design process a team needs (a strategy question for a different conversation), and it isn't for a team wondering whether the tests their AI assistant just wrote are testing anything real. Different problem. This one is vocabulary and arithmetic: what pairwise testing is, what it buys, what it gives up, and the one number, cost per case, that decides whether it's worth the trouble.
How pairwise testing works: the arithmetic
Take a checkout with six configuration parameters, four values each: payment provider (Stripe, Adyen, PayPal, Braintree), region (US, EU, APAC, LATAM), currency (USD, EUR, GBP, JPY), cart type (single-item, multi-item, subscription, mixed), device (desktop, iOS, Android, tablet), auth state (guest, registered, SSO, expired session). An ordinary shape for a real checkout, and already a combinatorial explosion too big to test exhaustively.
The full cross product is 4^6: 4 x 4 x 4 x 4 x 4 x 4 = 4,096 distinct configurations. Nobody runs the checkout 4,096 times, nor should they have to.
Pairwise testing asks a narrower question: cover every pair of values at least once, not every combination. Six parameters give C(6,2) = 15 parameter pairs, each with 4 x 4 = 16 value combinations, so a genuine all-pairs set has to hit 15 x 16 = 240 required pairs, no more and no fewer.
We handled pairwise test case generation with a greedy set-cover script (the same family of algorithm PICT and ACTS use, with far less engineering behind ours), then a second, independent script that enumerated all 240 required pairs and checked, row by row, that every one is covered before publishing a number. That check is what most write-ups of pairwise testing skip: several published "here's your pairwise set" examples turn out to be missing a pair once you actually verify them.
Here is that set:
| Payment | Region | Currency | Cart type | Device | Auth state |
|---|---|---|---|---|---|
| Stripe | US | EUR | Mixed | Desktop | SSO |
| Braintree | APAC | EUR | Subscription | Tablet | Expired session |
| PayPal | LATAM | GBP | Multi-item | iOS | Registered |
| Stripe | EU | GBP | Single-item | Android | Expired session |
| Adyen | EU | USD | Subscription | Desktop | Guest |
| Adyen | APAC | JPY | Single-item | iOS | SSO |
| Braintree | US | JPY | Multi-item | Android | Guest |
| PayPal | EU | JPY | Mixed | Tablet | Registered |
| PayPal | LATAM | USD | Subscription | Android | SSO |
| Braintree | LATAM | USD | Mixed | iOS | Expired session |
| Stripe | APAC | USD | Multi-item | Tablet | Registered |
| Adyen | LATAM | EUR | Single-item | Tablet | Guest |
| PayPal | APAC | GBP | Mixed | Desktop | Guest |
| Adyen | US | GBP | Subscription | Tablet | Registered |
| Adyen | LATAM | JPY | Multi-item | Desktop | Expired session |
| Braintree | EU | EUR | Single-item | Desktop | Registered |
| Stripe | LATAM | JPY | Subscription | iOS | Guest |
| PayPal | US | EUR | Single-item | iOS | Expired session |
| Braintree | EU | GBP | Multi-item | iOS | SSO |
| Adyen | APAC | EUR | Mixed | Android | Registered |
| Adyen | US | USD | Single-item | Tablet | SSO |
| PayPal | APAC | EUR | Multi-item | Tablet | Expired session |
Twenty-two rows out of 4,096 is a 99.46% reduction, while still guaranteeing every parameter's every value pairs with every other parameter's every value somewhere in the set.
The diagram below is that guarantee drawn out cell by cell. All 240 required pairs are grouped into the 15 parameter pairs they belong to, 16 value combinations each, and each square lights up on the first of the 22 rows that covers it. Watch where the returns fall off: rows 1 through 7 each cover 15 brand new pairs, because at that point nothing has been covered yet, and by row 22 there is exactly one pair left in the whole matrix that no earlier row reached.
What pairwise testing buys and what it gives up
The reduction only makes sense if most real defects don't need all six parameters to misbehave at once. That claim isn't folklore: NIST's combinatorial testing research (Kuhn, Kacker and Lei, "Practical Combinatorial Testing," NIST Special Publication 800-142) reports fault data from a NASA application where 67% of failures were triggered by a single parameter value acting alone, 93% by some 2-way interaction, and 98% by a 3-way interaction. The publication states its own interaction rule directly: most failures come from single-factor faults or the combined effect of two factors, with progressively fewer caused by three-way, four-way, and higher-order interactions. That's the basis for why covering every pair, not every combination, catches most real defects.
It's not all of it, and NIST says so plainly: pairwise testing can still miss 10% to 40% or more of a system's bugs, which is why the same publication won't call it sufficient on its own for mission-critical software. Our 22-row set proves the gap directly, not by citation. PayPal, APAC, and JPY never appear together in one row, even though PayPal pairs with APAC somewhere, PayPal pairs with JPY elsewhere, and APAC pairs with JPY in a third row. Each pair is covered; the specific three-way combination is not. JPY is a zero-decimal currency (no cents), and a rounding bug in PayPal's APAC settlement path might only surface when all three conditions hold at once. A pairwise set can't promise that row exists: covering every three-way combination is a bigger problem, C(6,3) = 20 triples times 4^3 = 64 values is 1,280 required triples, over five times the 240 pairs, a guarantee this technique was never built to give.
Stated plainly: pairwise testing buys most one- and two-parameter defects for a fraction of the cases, and gives up guaranteed three-way-plus coverage, betting that the residual risk is rare enough to accept.
Orthogonal array testing vs. pairwise testing
Orthogonal array testing gets used as a loose synonym for pairwise testing, and that's not quite right. An orthogonal array is a stronger, older requirement inherited from design of experiments: every pair of values across two parameters must appear the same number of times across the whole array, not just once. That balance is what a statistician needs to estimate how much each factor contributes to an outcome, separating a strong effect from a weak one with confidence, the right tool for measuring a continuous effect size. Most software testing doesn't need that precision: it asks a yes-or-no question (does this combination trigger a defect), not a how-much one.
What all-pairs generators like PICT and ACTS actually produce is a covering array: every pair of values appears at least once, no balance required. Dropping balance is why a covering array can be smaller than a comparable orthogonal array, and why our 22-row set works for six parameters when a classical 4-level orthogonal array can't. The standard construction that gives a balanced design in 16 runs at four levels each tops out at five factors: OA(16, 4^5, 2). Add a sixth 4-level parameter, as our matrix does, and no orthogonal array of that shape exists at 16 runs. That's exactly why our set needed 22 rows and not 16: the extra rows are the price of covering a sixth factor without the balance an orthogonal array insists on. If your matrix fits an existing orthogonal array, it also satisfies all-pairs coverage; past that shape, you need a covering array, which is what "pairwise testing" almost always means in practice.
For the other six test design techniques and where each fits, see the test design techniques overview rather than re-deriving all seven here: decision table testing is the right technique when the rules themselves are the test, and state transition testing is the right one when order matters more than combination. Deriving the partitions behind a matrix like this one is a separate technique, equivalence partitioning, with its own argument to make.
How Autonoma changes the cost per case
None of the arithmetic above solves the operational problem: generating a 22-row set is the easy part. Someone still has to run all 22 against a real build, seed account state for "expired session" or "SSO," drive UI paths across four devices, and tell a real defect from a flaky click. The matrix doesn't hold still either: add a seventh payment provider or drop a currency, and the whole covering array needs regenerating and re-verifying, or it quietly stops covering what it claims to.
That's the layer we built Autonoma to remove. Our platform reads your application's routes, components, and flows, plans test cases directly against them, and generates the endpoints needed to put your database in the right state for a given row (an expired session, a specific auth provider) instead of leaving that setup as a manual fixture. It executes cases against a real, live preview environment and classifies what comes back as a genuine bug rather than a flaky run, verifying findings against the actual running application, a grey-box approach that doesn't require hand-writing the runner. Keeping a covering array aligned automatically as parameters change, so a new payment provider in the code doesn't leave configuration coverage silently stale, is the Diffs Agent capability we're building next, not something the platform does today.
With generation and maintenance handled, what's left is what this article is actually about: how long each case takes to run, the decision rule's real input. To be clear about scope: if your six parameters are arguments to a pure function rather than states of a UI, a property-based or table-driven test in your own unit test framework is the right tool, not Autonoma. Autonoma is the behavioral, web end-to-end layer; it doesn't replace a unit test runner, and a six-argument pure function belongs in one.
The decision rule: multiply by cost per case
Here's the rule: multiply case count by measured cost per case, for both the full matrix and the pairwise set, against a wall-clock budget you actually care about.
At milliseconds per case (a pure function, a parser, a unit-level call) the full matrix answers itself: at 5 milliseconds each, 4,096 cases take about 20 seconds. Run all of them. The generator, its own maintenance cost, the review question of "why these 22 rows," and the three-way gap covered above are pure cost for no benefit at that speed. Skip the technique and test everything.
At minutes per case, a real browser against a real environment paying for login, navigation, and network calls, the arithmetic flips. The full 4,096 at 2 minutes each is over 136 hours sequential, not a suite anyone runs per pull request or at all. The pairwise set at 2 minutes each is 44 minutes: the difference between a suite that runs on every PR and one that runs overnight, if it runs in complete form at all. Here the reduction is the entire reason the coverage exists.
The crossover is arithmetic, not opinion, and it runs on your own numbers. Pick a wall-clock budget B (a CI timeout, whatever your team enforces) and your measured cost per case c. The full matrix stays viable exactly while total_cases x c <= B. With 4,096 cases, a 10-minute (600 second) budget holds only while c stays under roughly 146 milliseconds; the pairwise set, at 22 cases, has room down to over 27 seconds before hitting the same ceiling. Parallel workers divide the wall clock, not the total compute or the failures someone triages, so parallelism moves the crossover without erasing it. The fixed setup every row pays (spinning up an environment, logging in, seeding state) is why browser-driven cases land in the minutes bracket to begin with.
Both halves of that comparison move in different directions as execution keeps getting cheaper. The selection half, picking which 22 rows out of 4,096, is a budget optimization that loses value once the budget stops being scarce; as cost per case falls, more of the matrix moves into "run everything" on its own, no algorithm required. The oracle half, knowing what the checkout should actually do for a PayPal customer in APAC paying JPY with an expired session, doesn't get cheaper just because execution did. That's the part of test design that survives regardless of row count, and it's not something test case prioritization solves either: prioritization orders existing tests when time is short, pairwise testing decides which configurations deserve a test case at all.
Pairwise testing isn't obsolete. When each case is a real environment, a real login, a real network call away from an answer, a verified 22-row set instead of 4,096 is still one of the better trades available. What's changed is that its value is now a number computed for your own matrix and your own CI budget, not a rule of thumb applied regardless of what a case actually cost your team. Once cost per case is closer to minutes than milliseconds, the practical question stops being generation and becomes execution: who actually runs the 22 rows, on every PR, as the matrix keeps changing. That's the question Autonoma exists to answer.
Frequently Asked Questions
Pairwise testing, also called all-pairs testing, is a test design technique that generates a set of test cases covering every possible pair of parameter values at least once, instead of testing every possible combination. On a matrix with several multi-value parameters, it produces far fewer test cases than exhaustive testing while still exercising every two-parameter interaction at least once.
It replaces full combinatorial coverage with pair coverage. Instead of one test case per combination of all parameters, an algorithm generates the smallest set it can find where every pair of values across every two parameters appears in at least one row. On a 6-parameter, 4-value-each matrix this drops 4,096 possible combinations to a verified 22 rows, because covering every pair requires far fewer cases than covering every combination.
Orthogonal array testing is a stricter, older technique from design of experiments where every pair of parameter values must appear the same number of times across the whole array, not just at least once. That balance lets you estimate how much each factor contributes to an outcome, which matters for statistical experiments but is rarely needed for finding software defects. What most pairwise testing tools actually generate is a covering array, which only requires each pair to appear once, not a balanced orthogonal array.
Pairwise testing only guarantees coverage of one- and two-parameter interactions. It provides no guarantee for defects that only appear when three or more parameters take specific values at the same time, since a pairwise set can cover every individual pair inside a triple without ever placing all three values together in the same row. Published NIST research suggests pairwise coverage alone can still miss 10% to 40% or more of a system's bugs, which is why it is not recommended as the only testing performed on mission-critical software.
Skip it when running every combination is already cheap. If a test case executes in milliseconds, such as a unit-level check against a pure function, running the full combinatorial matrix costs less time than generating, reviewing, and maintaining a reduced set. Pairwise testing earns its complexity when each case is expensive to run, such as a browser-driven end-to-end test against a real environment, where the difference between a few dozen cases and several thousand decides whether a suite can run at all.
Yes, when the parameters describe states of your application's UI, like the checkout example in this article. Autonoma reads your codebase, plans test cases for the rows you specify, and runs them against a live preview environment, which is exactly the cost-per-case bottleneck this article's decision rule is about. It is not the right tool if your parameters are arguments to a pure function with no UI involved; that case belongs in your own unit test framework, not in a browser-driven suite.
