Using dif skills for agent-driven experimentation

dif init installs more than a folder. It drops four Claude Code skills into .claude/skills/ and writes a guidance block into your agent config files, so a coding agent can draft, edit, conclude, and triage experiments next to you. The skills read dif/context.json and the surface logs on the way in, so prior learning travels with the work.

What dif init put in your repo

Alongside the dif/ workspace, dif init writes:

.claude/skills/
  dif-author-experiment/     # draft and iterate on an experiment
  dif-conclude-experiment/   # write the decision, archive, learn
  dif-generate-surfaces/     # propose the surface set from the codebase
  dif-triage-experiments/    # act on Cloud verdicts: ramp, turn off, conclude
CLAUDE.md                    # a managed dif block
AGENTS.md                    # the same block, for other agents
.github/copilot-instructions.md # the same block, for Copilot
.cursor/rules/dif.mdc        # the same guidance, as a Cursor rule

dif owns only its marked block in CLAUDE.md, AGENTS.md, and .github/copilot-instructions.md, so anything else you keep there survives a re-run. To skip all of it, pass dif init --agents none (the deprecated --no-agent-files alias still works too). The CLI, the SDK, and the build work the same without it.

context.json: the session-start read

dif build regenerates dif/context.json. It lives outside the gitignored generated/ folder, so it is committed, and any agent that opens the repo sees the current state. Think of it as a CLAUDE.md for your experiments:

{
  "active": [
    { "id": "checkout-cta-v2", "surface": "checkout",
      "variants": ["control", "variant_a"], "running_for_days": 7 }
  ],
  "surfaces": [
    { "name": "checkout",
      "recent_learning": "Get it today lifted conversion 2.1% on returning" }
  ],
  "conventions": ["Fire exposure at render, never at assignment."]
}

The file carries what is active, on which surface, for how long, each surface's most recent learning, and the project conventions. Because the skills read it first, a new draft starts from what the last test found instead of from a blank page.

The four skills

Each skill is a plain SKILL.md with a trigger description. Your agent loads one when your request matches: say "draft an experiment" and dif-author-experiment answers. Three follow the daily loop. The fourth works across every running experiment at once. The skills reference covers each one in full.

Two of them read results from Dif Cloud and need a read token, through the MCP server or in DIF_TOKEN: see MCP and API. Without Cloud, the conclude skill asks you for the numbers and the triage skill stops.

dif-author-experiment

For drafting, editing, or fixing an experiment. It reads three files before writing anything: dif/context.json for what is already running, the surface's ## Learnings log so it does not re-test a settled question, and dif/config.yaml for the audience attributes it may use. Then it runs the loop:

$ dif new <id> --surface <surface>
# fill in the hypothesis, brief, rationale, audience, metrics
$ dif validate
$ dif build

It fills the stubs with opinions: one falsifiable hypothesis, variants that total 100 with control first, a single primary metric, and predicates only over declared attributes.

dif-conclude-experiment

For closing a test that has run. The decision you write becomes the next line in the surface log, and the next draft on that surface reads it. So the skill gets the numbers first: it calls get_experiment_results on Dif Cloud, or asks you when Cloud is not connected. It concludes only when the verdict is ship, turn_off, or conclude_inconclusive, unless you override it in chat. It starts from the one-line decision_draft Cloud writes and keeps its numbers.

$ dif conclude checkout-cta-v2 \
    --decision "Shipped variant_a. +2.1% completed_checkout (95% CI +0.8% to +3.4%), p=0.004 over 14d. Guardrails: refund_rate clean."

That command moves the file to concluded/, fills the ## Decision block, and appends a dated line to the surface log. A useful decision reads like the one above. "Inconclusive." tells the next draft nothing.

dif-generate-surfaces

For a fresh repo, where dif init left a single surface. A surface is a place you would run a test, not a route or a component, so most apps have three to eight. The skill reads your README, your routes, and your config, proposes a list with a one-line reason for each, and waits for your yes before it writes the files.

dif-triage-experiments

For the question "what needs action?" across every running experiment. It reads the Dif Cloud overview, shows one row per experiment with its verdict and next action, and acts on the next action: it edits weights for a ramp or a kill, and hands a conclusion to dif-conclude-experiment. Broken data and metrics with no direction are reported, never acted on. It confirms each change with you by default. On an unattended run, it opens one pull request per change, pushes at most 5 branches per run, and never merges.

A session looks like this

You talk to the agent the way you would a teammate. To start a test:

Draft an experiment on checkout testing a more urgent CTA, for returning visitors.

It reads the checkout log, runs dif new, fills the frontmatter, and validates. When the numbers are in:

Conclude checkout-cta-v2.

With Dif Cloud connected, it pulls the result, checks the verdict, writes the decision from the draft and asks you to confirm it. Then it archives the file, appends the learning, removes the call sites that W001 names and commits. You review the commit before you push. To check everything at once:

What needs action across my experiments?

It shows each experiment's verdict and proposes the ramps, kills, and conclusions, one change at a time.

The skills are an accelerant, not a requirement. The CLI and SDK behave the same without them, and the AGENTS.md, .github/copilot-instructions.md, and .cursor/rules/dif.mdc guidance points non-Claude agents at the same workflow. Once a test ships, the next step is a team-wide live read, which is what Dif Cloud adds.