A/B testing for developers: experiments in code

A/B testing for developers means running experiments in your codebase instead of a visual editor: you split traffic in code, assign each user deterministically, measure a primary metric, and conclude with a decision. The experiment is defined where the change is, not in a separate dashboard someone edits.

Most A/B testing tools were built for a marketer. They ship a WYSIWYG editor that rewrites the page in the browser after it loads, which flickers, slows the page, and does not fit a codebase where a change is a pull request and a deploy. A developer wants the experiment in the same place as the code, assigned on the server, and concluded with a number.

This post covers how an A/B test actually works in code: the lifecycle, how assignment avoids flicker, server-side versus client-side, and the parts people get wrong, which are significance and guardrails.

Key Takeaways

A/B testing for developers means the experiment lives in code, split and measured in the deploy pipeline, not a browser-based visual editor.

An A/B test is a feature flag held at a split with a hypothesis and a primary metric. Same mechanism, different intent.

Assign users deterministically by hashing the user id, so nobody flips between variants across page loads and no network call is needed to decide.

Server-side assignment avoids the flash of original content that client-side editors cause, and it works with server-side rendering.

Pick one primary metric and a guardrail before you start, and do not stop the test the first time it looks significant.

What A/B testing for developers means

An A/B test splits users into two or more groups, shows each a different variant, and measures which one moves a metric. “For developers” means that split lives in your code and your deploy pipeline, not in a hosted editor a non-engineer points at your live site.

The difference is not cosmetic. A visual A/B testing tool injects JavaScript that rewrites the DOM after the page loads, so the original renders first and then flips to the variant. A developer running the test in code decides the variant before render, ships it through review, and never paints the wrong thing. The experiment is versioned with the feature it tests.

An A/B test is a feature flag held at a split

An A/B test and a feature flag are the same mechanism. A flag decides which code path a user gets. An experiment does exactly that, then holds the split steady, states a hypothesis, and measures a metric until the numbers answer.

In dif they are the same file. A flag is one you are ramping toward 100%. An experiment is one you hold at 50/50 with a hypothesis and a primary metric, then conclude. If you are new to the flag side, what feature flags are covers it.

---
id: checkout-cta-copy
status: active
owner: sam@acme.com
surface: checkout
hypothesis: >
  A benefit-led CTA ("Get it today") lifts completed checkouts
  over the control ("Buy now").
variants:
  - id: control
    weight: 50
  - id: get_it_today
    weight: 50
metrics:
  primary: completed_checkout
  guardrails:
    - refund_rate
---

## Brief

Ship at 50/50. Conclude when completed_checkout reaches
significance and refund_rate holds.

Because a flag and an experiment share a format, an experiment that wins becomes a flag you ramp without touching application code. You change the weights, not the call site. That is experiments as files, the same idea as flags in the repo.

How to run an A/B test in code

The lifecycle is five steps, and most of the discipline is in the first and the last.

Write a hypothesis. One sentence: the change, the metric you expect it to move, and the direction. “A benefit-led CTA lifts completed checkouts.” A test without a hypothesis is just a coin flip you are paying attention to.
Split traffic. Hold the variants at a fixed ratio, 50/50 to start, and gate the code path on the assignment.
Assign and expose. Each user is bucketed to a variant, and an exposure event fires the first time they see it.
Measure the primary metric. Track the outcome and compare variants, watching the guardrail so a win on the primary does not hide a regression elsewhere.
Conclude. When the result is significant, record the decision and a one-line learning, then ramp the winner or drop it.

In code, steps 2 through 4 are one call site and one track call:

if (dif("checkout-cta-copy") === "get_it_today") {
  return <Cta variant="get_it_today" />;
}
return <Cta variant="control" />;

// when the user completes checkout:
dif.track("completed_checkout");

Step 5 is a command. dif conclude dates the decision, archives the file, and writes the learning back to the surface, so the next person who runs a checkout test reads what already happened. Concluding is the step teams skip, which is how experiment results end up lost in an old Slack thread.

Assignment without flicker

The part developers care about is how a user gets a variant. Do it wrong and users flip between control and treatment across page loads, which corrupts the data and looks broken.

The fix is deterministic assignment: hash a stable user id and map it into a fixed set of buckets, so the same user always lands in the same variant. dif computes the first four bytes of SHA-256(salt || user_id) mod 10,000 and reads the variant off the weights. Nothing is stored, there is no network call to decide, and a per-experiment salt keeps two tests from correlating.

Where you assign matters too. A client-side visual editor decides in the browser after load, which causes the flash of original content and misses users who block its script. Assigning on the server, or at the edge, decides before render. dif’s SvelteKit and React adapters do server-load assignment, so the right variant is in the first byte of HTML, with no flicker and nothing for an ad blocker to catch.

Guardrails, significance, and what to test

Two mistakes sink more A/B tests than any tooling problem, and both are about discipline, not code.

The first is stopping early. An A/B test wanders in and out of “significant” while it runs, and if you stop the moment it looks like a win, you ship noise. Evan Miller’s how not to run an A/B test explains why: decide the sample size or the duration up front, and hold to it. Most tests need more traffic than you expect, and a low-traffic surface may never reach significance at all.

The second is measuring one number. A lift in clicks that also lifts refunds is not a win. Pick a primary metric and at least one guardrail before you start, the way trustworthy experiment practice recommends, so a regression cannot hide behind the headline. In dif the guardrail is a line in the file.

And not everything is worth testing. An A/B test costs traffic and time, so spend it on decisions that matter and are hard to reverse, not on the color of a button nobody clicks.

Getting started

You can run a developer A/B test with any tool that assigns deterministically and lets you track an outcome. If you want the experiment in the repo, reviewed in a pull request and concluded with a recorded decision, dif does that. Install the CLI and scaffold a project:

npm install -g @dif.sh/cli
dif init

dif new drafts an experiment file with that surface’s recent learnings pre-loaded, dif validate checks it in CI, and dif build compiles it. The first experiment guide walks through one end to end.

A developer A/B test comes down to four things: a hypothesis in a file, a deterministic split in your code, a metric you agreed on before starting, and a recorded decision when the numbers land. The dashboard is the optional part.

FAQ

What is the difference between a feature flag and an A/B test? Intent, not mechanism. A feature flag decides which code path a user gets; an A/B test is a flag held at a fixed split with a hypothesis and a metric, run until the result is significant. Same assignment, same call site. In dif they are the same file.

How do you run an A/B test as a developer? Write a hypothesis, hold two variants at a fixed split, assign each user deterministically, track the outcome metric, and conclude with a decision when the result is significant. The split and the tracking live in your code, not a visual editor.

What is server-side A/B testing? Deciding a user’s variant on the server, or at the edge, before the page renders, instead of rewriting the page in the browser after it loads. It avoids the flash of original content, works with server-side rendering, and is not blocked by ad blockers.

What is a holdout experiment? A holdout keeps a small group of users on the old experience while everyone else gets the new one, so you can measure the long-run effect of a change you have already shipped. It is the same file format as any other experiment, with the split skewed and held.

How long should an A/B test run? Long enough to reach the sample size you decided on before starting, usually a minimum of one to two full business cycles (often two weeks) to cover weekly patterns. Do not stop the first time it looks significant, because that is how you ship noise.

Do you need a dashboard to run an A/B test? No. You need deterministic assignment, a tracked metric, and a way to judge significance. Analysis can run in the repo, through a stats library, or in a hosted layer like dif Cloud, but the experiment itself is defined in code.