Running concurrent A/B tests without mixing results
Running concurrent A/B tests on the same page is safe only when no user can land in two tests at once. Without that guarantee, two experiments compete for the same conversion, and a metric move can’t be attributed to either one.
Most teams hit this by accident. Two experiments target checkout at the same time: one swaps the CTA copy, the other changes how tax is displayed. Both roll out to 100% of visitors. A user can get get_it_today from the first test and tax_inclusive from the second, and if completed_checkout moves, neither experiment owner can say which change did it, or whether the combination did something neither would alone.
This post covers when concurrent A/B tests are actually safe to run, what an exclusion group does about the unsafe case, and how to catch the collision before it ships instead of after the numbers come back unreadable.
Key Takeaways
- Two A/B tests on the same surface are safe together only if their audiences are disjoint or they share an exclusion group, so one of them gives way.
- dif’s collision check (E007) fails a
dif validaterun when two active experiments target the same surface with overlapping audiences and no sharedexclusion_group.- An exclusion group orders member files by earliest
createddate; the first matching audience wins, and every other member in the group shows its default variant and records nothing for that user.- Excluding a test from another’s traffic shrinks its effective sample size. The group is a priority order, not a traffic split: if the older test has no
audiencerestriction it claims every user, and the newer one gets none until the older test concludes or is narrowed.- The fix is adding a matching
exclusion_groupvalue to each file.dif validateclears once both are tagged.
Running two A/B tests at once can contaminate both
Two experiments collide when they can both assign the same user and both move the metric you’re reading. That’s the whole condition. It doesn’t require the same file, the same owner, or even the same team.
Take the checkout example. checkout-cta-copy.md tests a benefit-led CTA against the control. checkout-price-display.md, opened by a different owner three weeks later, tests showing tax inline against showing it at the cart step. Neither file references the other. Both target checkout at 100%, because nobody checked what else was already live there.
A visitor can now land in four cells: control/cart-tax, control/inline-tax, get_it_today/cart-tax, get_it_today/inline-tax. If completed_checkout climbs 2%, that lift belongs to some combination of two changes, not to either experiment’s hypothesis. Worse, the two changes might interact: a clearer CTA next to a surprising tax line could move checkout completion in a way neither change produces by itself. That’s a textbook interaction effect, and reading it off two independent single-factor tests is exactly how it gets missed.
When concurrent A/B tests are safe to run together
Concurrent A/B tests are safe exactly when the two tests can’t both assign a user who converts on the metric you’re measuring. That happens two ways.
The first is disjoint audiences. A pricing test scoped to country: US and an onboarding test scoped to plan: trial don’t overlap if no user is both. Nothing needs to change; the populations are already separate, and each test reads its own metric cleanly.
The second is a shared surface with overlapping audiences, which is the checkout case above. Here the tests can’t both run on the full population, so one of them has to give way to the other, and that tiebreak is what an exclusion group decides.
A factorial experiment design, where you deliberately cross two factors and measure the interaction, is a third option, but it’s a different experiment with its own cell sizes and its own hypothesis. It’s not the default outcome of two unrelated tests that happened to launch in the same sprint.
Exclusion groups: one file, one variant per user
An exclusion group is a shared value in each experiment file’s frontmatter. Tag both files with the same exclusion_group, and dif guarantees each user participates in at most one of them.
---
id: checkout-cta-copy
status: active
owner: sam@acme.com
surface: checkout
hypothesis: >
A benefit-led CTA ("Get it today") lifts completed checkouts
over the control ("Buy now").
created: 2026-08-04
exclusion_group: checkout-page
variants:
- id: control
weight: 50
- id: get_it_today
weight: 50
metrics:
primary: completed_checkout
---
---
id: checkout-price-display
status: active
owner: priya@acme.com
surface: checkout
hypothesis: >
Showing tax inline lifts completed checkouts over showing it
at the cart step.
created: 2026-08-25
exclusion_group: checkout-page
variants:
- id: cart_tax
weight: 50
- id: inline_tax
weight: 50
metrics:
primary: completed_checkout
---
Members are ordered by earliest created date. checkout-cta-copy was created first, so it gets first claim on every visitor whose audience it matches. checkout-price-display only sees the users checkout-cta-copy didn’t claim. With no audience block on checkout-cta-copy, as written here, that is nobody: checkout-price-display gets no traffic until checkout-cta-copy concludes or its audience is narrowed. A user who isn’t assigned into checkout-price-display sees its default variant and fires no exposure event for it, so the data stays clean instead of counting a non-participant as a control.
Catch the collision in CI, not in the results
The frontmatter fix only works if someone remembers to add it before both files ship, which is exactly the step that gets skipped when two owners don’t know about each other’s test. That’s what dif validate’s E007 check is for: it scans active experiment files for any pair that targets the same surface with overlapping audiences and no shared exclusion_group, and fails the run.
$ dif validate
error E007 dif/experiments/active/checkout-cta-copy.md:1:1: experiments `checkout-cta-copy` and `checkout-price-display` both target surface `checkout` without a shared exclusion_group, and their audiences are not provably disjoint
help: Add the same `exclusion_group:` value to both, narrow one of the audiences so they're provably disjoint, or change one of their `surface:` fields.
✗ 1 error(s), 0 warning(s)
Add the exclusion_group line above to each file and the same command passes. The collision fails the pull request like a broken build, before either test’s numbers have a chance to get tangled, instead of surfacing three weeks later as an unexplained metric move someone has to debug by hand.
The cost of excluding a test from the pool
Excluding a test from another’s traffic has a real cost. checkout-price-display only gets the traffic checkout-cta-copy leaves behind, so its effective sample size is smaller than it would be running alone, and possibly smaller than the sample size you planned for when you scoped the test. While checkout-cta-copy targets everyone, that leftover is zero.
State that up front rather than discovering it at the significance check. If a surface already has a long-running experiment claiming most of its traffic, a new test entering the same exclusion group should expect to run longer, or should be sized against the smaller pool it will actually get, not against the full surface. The alternative, running both unconstrained, produces a faster answer to a question you can’t actually answer. To give both tests traffic at the same time, scope them to audiences that can’t overlap instead, such as device_type: mobile on one and device_type: [tablet, desktop] on the other. That passes dif validate without a group.
Getting started
If your experiments already live as files, adding an exclusion group is a two-line edit and a dif validate run.
npm install -g @dif.sh/cli
dif init
dif new scaffolds an experiment file on a surface; if another active file already claims that surface, dif validate tells you before you ship. The experiments as files model and the A/B testing for developers guide cover the rest of the lifecycle, from hypothesis to dif conclude. The format reference documents every frontmatter key, including exclusion_group.
FAQ
Can you run two A/B tests at the same time? Yes, if their audiences are disjoint or one is scoped out of the other’s traffic with an exclusion group. Running two tests on the same surface with overlapping audiences and no exclusion group means any metric move could belong to either test, or to both together.
What happens if two experiments target the same users? Whichever effect you measure can’t be separated between the two changes, and if the changes interact, the combined effect may not match either one’s individual lift. The fix is an exclusion group, not a bigger sample size.
What is an exclusion group in A/B testing? A tag shared across experiment files that guarantees each user is assigned to at most one member of the group. The earliest-created file gets first claim on a matching user; every other member shows its default variant to that user and records nothing.
How do you know if two experiments will interact? You often don’t until you’ve run a dedicated factorial test, which is more cells and more traffic than either test alone. Treating unrelated concurrent tests as if they can’t interact is the assumption that gets broken most often, not a rule that’s usually safe.
Does excluding a test slow it down? It can. A test that loses the tiebreak to an earlier-created one in its exclusion group only sees the remaining traffic, which is none while the earlier test targets every user, so it needs more time or a smaller expected effect to reach significance. Size the test against the pool it will actually get.
Can I run concurrent A/B tests without a tool that checks for collisions? Yes, by tracking every active test’s surface and audience in one place and checking manually before each launch. A validator that fails the build on overlap catches what a spreadsheet eventually misses.
Conclusion
Concurrent A/B tests are only as trustworthy as the guarantee that no user sits in two of them at once. Disjoint audiences give you that for free; a shared surface with overlapping audiences needs an exclusion group, which is two lines of frontmatter and an ordering rule, not a second system to run.
Three things worth keeping: check whether your tests’ audiences actually overlap before assuming they don’t, tag colliding tests with a shared exclusion_group so the earliest one wins the tiebreak and the rest record nothing for that user, and size a test against the traffic it will actually get once it’s excluded from another. dif validate catches the collision in CI, which beats catching it in the results.
Install the CLI, run dif validate against your active experiments, and see if any two are already colliding: npm install -g @dif.sh/cli, then dif init.