DEV Community: David Sert

Sample Ratio Mismatch: Is Your A/B Test Broken?

David Sert — Thu, 20 Aug 2026 05:10:45 +0000

Your experiment finished, the dashboard shows a winner, and something still feels off. The traffic split you set to 50/50 is sitting at 47/53. Before you write up the result or ship the change, you need to answer one question: is the split you observed close enough to the split you designed, or is the test itself broken? That question has a precise, statistical answer, and getting it wrong is one of the most common ways a team ships a change based on a number that was never real. This is sample ratio mismatch, and it is the first thing a senior experimenter checks before looking at any lift.

What sample ratio mismatch is

A sample ratio mismatch (SRM) occurs when the number of visitors actually assigned to each variation differs from the split you intended by more than random chance can explain. You designed a 50/50 test; the engine delivered something that a fair coin, flipped that many times, would essentially never produce.

The important word is chance. You should never expect a perfectly even split. If you assign 20,000 visitors with a fair 50/50 rule, landing on exactly 10,000/10,000 is rare — the count wobbles around the target every time, the same way 10,000 coin flips almost never give exactly 5,000 heads. SRM is not "the split isn't perfect." SRM is "the split is so far off that randomness is no longer a believable explanation."

That distinction is why you cannot judge SRM by eye. Looking at 47/53 tells you the direction and size of the gap but nothing about how surprising it is, because surprise depends on how many visitors produced that gap. The same 47/53 is unremarkable at 400 visitors and a five-alarm fire at 400,000. You need a test that accounts for sample size. The tool for that is the chi-square goodness-of-fit test, and OptiPilot's free SRM Checker runs it for you — paste in your observed counts and expected split, and it returns the p-value and verdict.

Why an SRM invalidates your results

Randomized assignment is the entire reason an A/B test can claim causality. When you split traffic randomly, the two groups are, on average, identical in every way — same mix of devices, same proportion of loyal customers, same intent — except the change you are testing. That balance is what lets you attribute any difference in outcomes to the variation rather than to a difference in who was in each group.

An SRM is direct evidence that the randomization did not work as designed. If a mechanism pushed extra visitors into one variation, that same mechanism almost certainly pushed a non-random kind of visitor. Redirects that time out drop the impatient and the slow-connection users. Bot filters that fire unevenly strip out a specific traffic source. In every case the groups are no longer comparable, so the lift you measured is a mix of the real treatment effect and a selection bias you cannot separate out.

This is the point that trips up teams under deadline pressure: you cannot adjust your way out of an SRM. There is no correction factor, no reweighting, no "the split was 45/55 so I'll scale the numbers." You do not know which visitors are missing or why, so you cannot model the bias. An SRM means the data is untrustworthy at the source. The only valid responses are to find and fix the cause, then rerun — or, if you can prove the cause is benign and non-differential, to document exactly why. Reading the Optimizely results page is meaningless until the split checks out.

How SRM is detected with a chi-square test

The standard test is Pearson's chi-square goodness-of-fit. It compares the visitor counts you observed in each variation against the counts you expected under your intended split, and returns a p-value: the probability of seeing a gap at least this large if assignment were truly random.

The statistic is:

chi-square = sum over each variation of (observed - expected)^2 / expected

degrees of freedom (df) = number of variations - 1

For a two-arm test, df = 1. For a three-arm 34/33/33 test, df = 2, and so on. You compute the statistic, look up the p-value for that value and df, and compare it to a threshold.

The industry alarm threshold is p < 0.001. If the p-value falls below it, you treat the experiment as having an SRM and stop trusting the results until you find the cause. Some teams run stricter: Microsoft's experimentation platform uses 0.0005, and a few use 0.0001. The threshold is deliberately far tighter than the 0.05 you use for a metric result, for two reasons. First, you run this check on every experiment, so a loose threshold would drown you in false alarms. Second, SRM is a binary data-integrity gate, not an effect you are trying to measure — you want it to fire only when something is genuinely wrong. Note that a real SRM usually produces a p-value orders of magnitude below any of these thresholds, not a marginal miss.

A worked chi-square example

Take a 50/50 test. You intended an even split; you observed 10,200 visitors in A and 9,800 in B, for 20,000 total. Under a fair split you expect 10,000 in each.

Observed:  A = 10,200   B = 9,800     Total = 20,000
Expected:  A = 10,000   B = 10,000    (50% of 20,000 each)

chi-square = (10,200 - 10,000)^2 / 10,000 + (9,800 - 10,000)^2 / 10,000
           = (200)^2 / 10,000 + (-200)^2 / 10,000
           = 40,000 / 10,000 + 40,000 / 10,000
           = 4 + 4
           = 8.0

df = 2 variations - 1 = 1
p-value for chi-square = 8.0, df = 1  ->  p ~= 0.0047

How to read that: p ≈ 0.0047 means a gap this large would happen about 5 times in 1,000 by pure chance. That clears the conventional 0.05 bar, so a naive check would panic. But it is above the 0.001 SRM threshold, so by the standard you do not declare an SRM. It sits in the uncomfortable middle — worth a glance, not worth halting for. This is exactly why the strict threshold exists: a 2% wobble at this sample size is not yet strong enough evidence of a broken split.

Now keep the same 51/49 ratio but run 10x the traffic:

Observed:  A = 102,000   B = 98,000    Total = 200,000
Expected:  A = 100,000   B = 100,000

chi-square = (2,000)^2 / 100,000 + (-2,000)^2 / 100,000
           = 40 + 40
           = 80.0

df = 1
p-value for chi-square = 80.0, df = 1  ->  p ~= 3.7e-19  (effectively zero)

Identical proportions, wildly different verdict. At 20,000 visitors the 51/49 split is plausibly noise; at 200,000 it is a certainty that something is wrong. This is the single most important intuition about SRM: the ratio alone tells you nothing — the mismatch is a function of the ratio and the sample size together. It is also why staring at the percentages on a dashboard will never substitute for the test.

What causes sample ratio mismatch

An SRM is a symptom, like a fever — it tells you something is wrong without telling you what. The KDD 2019 taxonomy from Microsoft and Booking.com researchers organizes the causes by where in the pipeline the visitor count gets distorted. Walking that pipeline is how you narrow the search.

flowchart LR
    A[Assignment] --> B[Exposure]
    B --> C[Logging]
    C --> D[Analysis]
    A -.->|"faulty hash, bad IDs,<br/>uneven ramp, carryover"| A
    B -.->|"redirect timeouts, flicker,<br/>variation errors, latency"| B
    C -.->|"bot filters, dropped events,<br/>caching, ad blockers"| C
    D -.->|"wrong trigger point,<br/>bad joins, segment filters"| D

Assignment-stage causes

The split is wrong at the moment of bucketing. A faulty randomization or hash function, corrupted or reassigned visitor IDs, or carryover effects from a previous experiment that still tag returning users. Uneven ramping belongs here too: if you launch variation B a day after A, or ramp them to different allocations, the counts diverge for a completely mundane reason.

Exposure-stage causes

Assignment was fair, but visitors are lost between being bucketed and being counted. Redirect tests are the classic offender — a visitor sent to a separate URL can close the tab before the redirect completes, so the event never fires and that visitor is never counted. Because the delay is not symmetric across variations, the counts drift. Client-side flicker and load races drop a variation when the page renders before the experiment code runs, latency in one variation lets impatient users leave first, and a variation that errors for a subset of browsers silently loses those visitors.

Logging and analysis-stage causes

The split was fine; the measurement distorts it. Bot and spam filtering applied unevenly strips a traffic source from one arm. Ad blockers and tracking-prevention block the logging beacon for some visitors more than others. Caching or CDN behavior serves stale assignments. On the analysis side, choosing the wrong starting point — counting from a downstream event that already differs between variations rather than from first exposure — manufactures an SRM out of clean data. Bad table joins and delayed log arrival do the same.

The most useful directional clue: excess visitors in the treatment arm often points to logging or engagement changes (the new variant generates more events, so more one-time visitors get recorded), while missing visitors in treatment points to exposure failures like redirects, errors, or latency.

How to diagnose an SRM

Once the check fails, resist the urge to guess. Diagnosis is a process of localization: find the segment or stage where the mismatch lives, and the cause usually becomes obvious.

flowchart TD
    S[SRM detected] --> Q1{Is it in every segment<br/>or just some?}
    Q1 -->|Isolated to one<br/>browser/device/source| Seg[Suspect that segment:<br/>rendering bug, bot source,<br/>ad blocker, redirect on that platform]
    Q1 -->|Spread evenly<br/>across all segments| Q2{Does the split break<br/>at assignment or later?}
    Q2 -->|Wrong at first exposure| Assign[Assignment layer:<br/>hashing, IDs, ramp timing,<br/>overlapping experiments]
    Q2 -->|Fine at exposure,<br/>off downstream| Q3{Redirect or heavy<br/>client-side render?}
    Q3 -->|Yes| Exp[Exposure loss:<br/>redirect timeouts, flicker,<br/>variation errors]
    Q3 -->|No| Log[Logging/analysis:<br/>bot filter, wrong trigger point,<br/>join error, caching]

Work through it in this order:

Segment the mismatch. Break the counts down by browser, device, operating system, day, traffic source, and new-vs-returning. If the SRM collapses into one segment — say, only Safari, or only mobile, or only the day of a deploy — you have found it. A rendering bug, a browser that rejects redirects, or a bot wave from one source will all show up as a single hot segment.
Find the stage where the ratio breaks. Compare the split at assignment, at exposure, and at analysis. If bucketing is even but exposure is skewed, the loss is happening after assignment (redirects, errors, latency). If assignment itself is off, the problem is upstream in the hashing or ID logic.
Check the analysis trigger. Confirm you are counting from first exposure, not from a downstream event that the treatment itself changed. This alone explains a large share of "SRMs" that are really analysis artifacts.
Line up the timeline. Map the onset of the mismatch against deploys, ramp changes, and marketing sends. SRM that starts precisely at a code push has a precise cause.

One caution specific to how detection works: run the SRM check on the whole experiment population, not on filtered segments as a decision input. Segmenting is a diagnostic tool for finding the cause once an SRM is flagged — it is not a place to hunt for fresh imbalances to act on, because slicing enough ways will always turn up a low p-value by chance.

How to prevent SRM

Prevention is cheaper than diagnosis. The teams that rarely see SRM have built these habits in:

Assign server-side where you can. Server-side or edge assignment removes the flicker, redirect, and client-render failure modes that cause most exposure-stage mismatches. Client-side tests are more exposed and need tighter QA.
Bucket at the point of exposure. Assign the visitor at the moment they are actually eligible to see the change, not at some earlier step that a variation may reach at a different rate. Assign-at-exposure keeps the counted population aligned with the treated population.
Run an [A/A test**](https://optipilot.com/data/aa-testing) first.** An A/A test — two identical variants — should show no SRM. If it does, your instrumentation is broken before you have tested anything real. It is the cheapest possible way to catch a pipeline problem.
Put an SRM check in CI and pre-analysis. Make the chi-square check an automated gate that runs continuously, not a manual step someone remembers at the end. Add it to your pre-launch experiment QA checklist so it is verified every time.
Be careful with redirects and uneven ramps. Prefer single-page variation over redirects when you can; when you cannot, expect and monitor a small imbalance. Ramp all variations together, and never adjust traffic mid-flight to "rebalance" counts — that corrupts the results further.

How Optimizely detects SRM automatically

Optimizely does not make you run this check by hand. Its Stats Engine A/B tests include automatic SRM detection with an Experiment Health indicator on the results page: a green Good status means no imbalance, and an SRM detected status means the test has failed a traffic check and needs investigation before you trust it.

One design detail matters for practitioners. Optimizely's detector does not use a one-shot chi-square test. It uses a sequential SRM (SSRM) method that checks traffic counts continuously as data arrives, so it can flag a problem within the first days of a launch — when stopping the test actually limits the damage — rather than only at the end. This is the same reasoning behind sequential testing and the peeking problem: a fixed-horizon test is only valid if you look once, whereas a sequential test is built to be monitored continuously. A retroactive end-of-experiment chi-square check has the opposite problem — by the time the totals even out, a transient mid-experiment SRM can be hidden entirely.

A few operational limits are worth knowing. Optimizely's automatic detection applies to Stats Engine A/B tests with Manual traffic distribution (not Stats Accelerator), running 45 days or less, with at least 1,000 visitors, and it evaluates only first decisions. It does not run on segmented results, and it does not check paused or archived experiments. So the manual chi-square check and the SRM Checker still earn their place: for Stats Accelerator tests, for segment-level diagnosis once an alarm fires, and for any experiment run on a platform without built-in detection.

Trust the split before you trust the lift

Sample ratio mismatch is the smoke detector of experimentation. It does not tell you what is on fire, but it tells you — reliably, and before you make an expensive decision — that you should not trust the numbers yet. The discipline is simple and non-negotiable: check the split with a chi-square test against a strict threshold, and if it fails, find and fix the cause rather than reasoning around it. A test with an SRM is not a weaker result; it is not a result at all.

Make it the first thing you look at, not the last. Before you read a single metric, run your observed counts through the SRM Checker. If the split holds, everything downstream — your lift, your confidence interval, your variance-reduced estimates — rests on solid ground. If it doesn't, you have saved yourself from shipping a decision built on a broken experiment.

Prevent Optimizely from being blocked by ad-blockers using AWS

David Sert — Thu, 20 Aug 2026 05:10:10 +0000

Category: Privacy

Some ad-blockers might block Optimizely from running. In this tutorial, we’ll see how we can prevent this from happening.

Ad-blockers will often look at the domain the request is originating from and if it happens to be from a list of known analytics tools (such as Optimizely.com), the ad blocker will block the network request from happening.

The key solution is to proxy requests via an API gateway which will forward all requests back to Optimizely.com.

Here’s how.

Step 1: Create an AWS API Gateway

flowchart TB
    subgraph Browser
        A[Your Website] --> B[Proxy Request]
    end

    subgraph AWS["AWS CloudFront"]
        B --> C[Your Domain cdn.yourdomain.com]
        C --> D[CloudFront Distribution]
    end

    subgraph Optimizely
        D --> E[cdn.optimizely.com]
        E --> F[Optimizely Snippet]
    end

    F --> A

    style AWS fill:#ff9900,color:#fff
    style Optimizely fill:#0037ff,color:#fff

Head the AWS Management Console then head to the API gateway section

Then, click on Create New API

Select HTTP API as the API type:

Step 2: Create the required routes

Now we are going to create our API routes and where the API will forward the requests to.

We’ll need 3 routes, depending on the Optimizely product you use:

One that allows us to retrieve the Optimizely snippet (if you are using Web). The snippet is fetched via a GET request done to https://cdn.optimizely.com. You can find the full snippet URL inside your Project settings.
One that allows us to send decision & conversion events (logx.optimizely.com). This route will be fetched via a POST request to https://logx.optimizely.com/v1/event.
One that allows us to retrieve the Optimizely datafile (if you are using Full Stack). This route will be fetched via a GET request to https://cdn.optimizely.com/datafiles. You can find your datafile URL inside your Project settings.

Here’s how it should look like once properly filled out:

AWS will ask you for a confirmation:

Change the ANY to be exactly the same as the method on the right-hand side. (POST, GET & GET)

Congrats you now have a working API which will proxy requests to Optimizely.com.

Step 3: Fetch Optimizely from your newly-created API gateway

Now that we’ve got a working API, it’s time to update our website to start fetching from this API.

Optimizely Web

Update your script tag that contains Optimizely to no longer fetch the file from cdn.optimizely.com but from your AWS API Gateway. You’ll find the invoke URL on the main API page, as such:

Now to ensure the Optimizely snippet sends events to the API gateway instead of the default Optimizely endpoint, this is a custom snippet setting that can’t be configured by a customer. You’ll need to ask your account manager about it. They can amend your snippet to ensure the snippet dispatches events to your newly-created API.

Optimizely Full Stack

For Full Stack, you’ll need to customize the SDK’s createInstance method to include a new datafile URL pointing to your API, as such:

const optimizely = require('@optimizely/optimizely-sdk');

const optimizelyClientInstance = optimizely.createInstance({
  sdkKey: '',
  datafileOptions: {
    autoUpdate: true,
    urlTemplate: 'https:///datafiles/%s.json',
  },
});

You’ll also need to provide a custom event dispatcher which will dispatch the events back to your newly AWS API Gateway. You can use this built-in event dispatcher and change line 39 to indicate the POST endpoint of your newly created API.

Related guides

Optimizely vs Amplitude: Experimentation Compared

David Sert — Thu, 20 Aug 2026 05:09:34 +0000

"Optimizely vs Amplitude" is a slightly unfair framing, because the two products start from opposite ends of the same problem. Optimizely is an experimentation platform that added analytics; Amplitude is a product analytics platform that added experimentation. Both can run an A/B test and tell you which variation won, but the path each takes — and the org each is built for — is different. This guide compares them honestly so you can pick the right tool for your situation, or decide to run both.

TL;DR of the difference

Optimizely leads with experimentation. Its core products — Web Experimentation (visual, client-side testing) and Feature Experimentation (server-side feature flags and SDK-based tests) — are purpose-built to run a high volume of rigorous experiments across marketing pages, product surfaces, and backend logic, with the Stats Engine making continuous monitoring statistically safe.

Amplitude leads with product analytics. Amplitude Experiment is an experimentation layer bolted onto a best-in-class behavioral analytics engine, so every test is automatically connected to deep funnel, retention, and cohort analysis on the same event data.

If your primary question is "what should we test, where, and how do we govern a large testing program," Optimizely is the stronger center of gravity. If your primary question is "how do users behave, and can I experiment without leaving my analytics," Amplitude is compelling. They are not mutually exclusive — many teams run both.

The decision comes down to two questions: what is the primary job, and who runs the tests?

flowchart TD
  A{What is the primary job?} -->|Run and govern experiments| B{Who builds most of the tests?}
  A -->|Deep behavioral analytics| C{Is Amplitude already your<br/>analytics system of record?}
  B -->|Marketers, in a visual editor| D[Optimizely Web Experimentation]
  B -->|Engineers, behind feature flags| E[Optimizely Feature Experimentation]
  C -->|Yes, flag-based product tests| F[Amplitude Experiment]
  C -->|No, and you test backend + marketing pages| G[Optimizely for experiments,<br/>Amplitude for analysis]

What each tool is built for

Optimizely

Optimizely splits experimentation into two complementary products. Web Experimentation uses a WYSIWYG Visual Editor and a JavaScript snippet, letting marketers and optimization teams build A/B tests, redirect tests, and multivariate tests on a live site without engineering for most changes (custom HTML/JS is available when needed). Feature Experimentation is developer-first: server-side and client-side SDKs (Java, Python, Go, C#, JavaScript/Node, Swift, Android, and more) wrap experiments in feature flags, so you can roll features out, roll them back instantly, and test backend logic such as algorithms, pricing, or APIs. Both share Optimizely's Stats Engine for results.

The platform is built for experimentation breadth and program scale: visual marketing tests and deep server-side tests under one roof, audience targeting, mutual exclusion groups, multi-armed bandits, and a free Rollouts tier for teams starting with feature flags.

Amplitude

Amplitude's foundation is its analytics graph: every event a user fires is stored and queryable through funnels, retention curves, pathfinder, and behavioral cohorts. Amplitude Experiment sits on top of that data, offering feature flags, server-side and client-side evaluation, and both feature-flag-based experiments and a web experiment capability. Its defining advantage is that experiment results read from the same event stream as the rest of your analytics — so a winning variation can immediately be sliced by any cohort or downstream behavior you already track.

Amplitude is built for product teams who live in their analytics and want experimentation to be a natural extension of behavioral analysis rather than a separate discipline.

A note on the moving market: OpenAI acquired the experimentation vendor Statsig in September 2025, and in May 2026 Amplitude announced it was taking over the Statsig brand, platform, and customer base. That leaves Amplitude with two experimentation lineages (its native Experiment product and the Statsig platform), and some analysts have flagged near-term uncertainty about how the overlapping capabilities will consolidate. Treat any roadmap promises in this area as provisional.

Head-to-head comparison

Dimension	Optimizely	Amplitude
Primary use case	Experimentation across web and product/backend	Product analytics, with experimentation layered on
Experimentation model	Visual (Web Experimentation) + feature flags / server-side SDKs (Feature Experimentation)	Feature-flag-based experiments + web experiment capability
Visual/no-code testing	Yes — mature WYSIWYG Visual Editor	Limited; beyond simple changes, work tends to need engineering
Server-side / SDK testing	Strong, broad SDK coverage	Yes, via Experiment SDKs
Statistics engine	Stats Engine (always-valid, sequential, controls false positives)	Sequential testing, t-tests, CUPED, multi-armed bandits
Analytics depth	Solid experiment analytics; warehouse-native analytics available	Best-in-class behavioral analytics, cohorts, retention, funnels
Integration model	Connects to external analytics (including Amplitude); CMS/DXP ecosystem	Experimentation tied directly to native event data
Pricing model	MAU-based (free Rollouts tier for flags)	Event-volume based
Ideal team	Optimization, growth, and engineering teams running a structured testing program	Product teams who already standardize on Amplitude analytics

Experimentation capabilities compared

The clearest difference is where and how you can test.

Optimizely covers two distinct surfaces well. For marketing and front-end teams, the Web Experimentation Visual Editor makes it straightforward to change copy, layout, and styling and ship an A/B test without a deploy. For engineering and product teams, Feature Experimentation runs tests behind feature flags directly in application code — including server-side paths where there is no DOM to manipulate, such as recommendation algorithms, checkout logic, or API behavior. Flags double as a kill switch, so a bad variation can be turned off remotely without redeploying. Optimizely also supports targeted rollouts, mutual exclusion, and multi-armed bandit optimization.

Amplitude Experiment is primarily flag-based: you gate a feature behind a flag, evaluate it client- or server-side, and measure the result against your event data. This is a clean model for product experimentation and progressive rollouts. Its web experimentation capability exists, but for anything beyond simple changes, practitioners generally report that meaningful variations require engineering involvement rather than a marketer-friendly visual workflow. If a large share of your testing is on marketing pages built by non-engineers, that distinction matters.

On the statistics, both are credible. Optimizely's Stats Engine is designed for "always-valid" inference — you can monitor results continuously without inflating false-positive rates, which suits teams that watch dashboards daily. Amplitude likewise uses sequential testing for valid-anytime results, and supports t-tests, CUPED for variance reduction, multi-armed bandits, mutual exclusion groups, and holdouts. Neither team should feel they are settling for weak statistics.

Analytics and data compared

This is where Amplitude's genuine strength shows. Because Experiment runs on the same event pipeline as Amplitude Analytics, a result is never a dead end: you can immediately ask "did the winning variation help retention at day 30," "how did it perform for this behavioral cohort," or "where in the funnel did the lift come from" without exporting data or stitching IDs across tools. For teams whose core competency is behavioral analysis, that tight loop is the main reason to consider Amplitude Experiment over a standalone testing tool.

Optimizely's analytics are good for reading experiment results and, increasingly, offers warehouse-native analytics that let you analyze decision and event data alongside the rest of your data in Snowflake, BigQuery, or Databricks. But product analytics is not its historical center of gravity the way it is Amplitude's. If your team's daily home is rich cohort and retention analysis, Amplitude will feel more native; if your team's daily home is running and governing experiments, Optimizely will.

A practical caveat on cost: Amplitude's event-volume-based pricing can scale quickly for high-traffic products, whereas Optimizely prices experimentation on monthly active users and offers a free Rollouts tier for feature flags. Model both against your actual traffic before deciding — the cheaper option depends heavily on your event volume versus user count.

When to choose Optimizely

Optimizely is the stronger choice when:

Experimentation is the primary job, not a feature of your analytics tool. You want one platform that handles both marketing-page tests and deep server-side product tests.
Non-engineers run a meaningful share of tests. The Visual Editor lets growth and marketing teams ship experiments without a deploy.
You test in the backend — algorithms, pricing, infrastructure changes — where flag-based, server-side experimentation and an instant kill switch are essential.
You run a high-volume, governed program and want mutual exclusion, bandits, audience targeting, and always-valid statistics in one place.
You are starting with feature flags and want a free on-ramp via Rollouts before scaling up.

When Amplitude makes sense

Amplitude Experiment is the better fit when:

Amplitude is already your analytics system of record and most of your team's decisions start in its funnels, cohorts, and retention reports.
Your experimentation is product-led and flag-based — gating features, progressive rollouts, and measuring impact on behavioral metrics.
The analysis loop matters more than test authoring breadth. You value being able to slice any result by behavioral cohort instantly, on the same data, without integration work.
You are not relying on a visual, marketer-driven editor for the bulk of your tests.

Be deliberate here given the current consolidation around the acquired Statsig platform: confirm which experimentation product a vendor is steering you toward and what its supported roadmap looks like.

How they work together

These tools are not an either/or for many organizations, and the most pragmatic answer is often "both." A common pattern is to run experiments in Optimizely and analyze them in Amplitude: Optimizely decides which variation a user sees and provides the statistical results, while the variation a user was bucketed into is sent into Amplitude as a user property or event, so you can analyze experiment impact against your full behavioral dataset.

This is a supported, documented integration. Optimizely Feature Experimentation can forward decision data to Amplitude via a decision notification listener (setting an [Optimizely] <flagKey> user property and an optional impression event), and Optimizely Web Experimentation integrates through custom analytics extensions, with the connector built and maintained by Amplitude. If your team has standardized on Amplitude for analytics but wants Optimizely's experimentation breadth and visual testing, this combination gives you the best of both rather than forcing a single choice.

The bottom line

Pick based on where your center of gravity sits. If experimentation is the discipline you are investing in — across marketing, product, and backend — Optimizely is built for that job and gives non-engineers and engineers a shared platform with rigorous statistics. If best-in-class behavioral analytics is your foundation and you want experimentation as a tightly coupled extension of it, Amplitude Experiment earns its place. And if you have both needs, the supported integration means you do not have to choose: experiment in Optimizely, analyze in Amplitude.

This site covers both Optimizely products in implementation depth. Browse the Feature Experimentation guides and the Web Experimentation guides, or start from the integration catalog. Not sure which platform fits your stack? Contact us.

Related guides

How the Optimizely Stats Engine Works

David Sert — Thu, 20 Aug 2026 05:08:58 +0000

Optimizely's Stats Engine is the statistical framework behind every winner, loser, and "inconclusive" verdict on the Experiment Results page. It exists to solve a specific, expensive problem: practitioners watch experiments in real time and make decisions the moment a result "looks significant," and traditional statistics punishes that behavior with a flood of false positives. Understanding how Stats Engine works is the difference between trusting a result and acting on noise.

What Stats Engine Is and the Problem It Solves

Stats Engine is Optimizely's proprietary statistical methodology for evaluating A/B tests. It is a frequentist engine built on sequential testing combined with false discovery rate** (FDR) control**. Those two mechanisms are the whole story, and each addresses a distinct failure mode of classical A/B testing.

The first failure mode is the peeking problem. Traditional fixed-horizon statistics (the t-test being the canonical example) are only valid if you commit to a sample size in advance, wait until you reach it, and look exactly once. Every time you check results early and react, you give random noise another chance to cross the significance threshold. A test that peeks repeatedly can report a false "winner" far more often than its stated error rate suggests. But waiting passively for a pre-computed sample size is exactly what real experimentation teams do not do.

The second failure mode is the multiple comparisons problem. Real experiments rarely track one metric on one variation. Add more variations and more metrics and the chance of at least one false positive climbs quickly, even though each individual test holds its error rate. Worse, the rate that actually matters to a decision-maker, the proportion of false positives among the results you act on, is higher still.

Stats Engine is engineered so that both problems are handled automatically. As Optimizely's documentation states, results are always valid: you can monitor a test continuously and stop as soon as you have a clear winner, without invalidating it, because the engine controls the false discovery rate throughout.

How Sequential Testing Solves the Peeking Problem

Sequential testing is the mechanism that makes continuous monitoring safe. Rather than computing a single p-value at one fixed endpoint, Stats Engine evaluates the experiment as evidence accumulates over time and produces inferences that remain valid no matter when you look.

Optimizely's own framing uses a baking analogy that is worth keeping in mind:

Fixed Horizon : Set a timer before baking. You may only open the oven
                when the timer ends. Open it early and the result is unreliable.
Sequential    : Put the cake in without committing to a time. Open the oven
                whenever you like to check; looking never ruins the result.
                When it looks done, it is done.

Mechanically, Stats Engine does not compute one confidence interval. It computes a series of 100 successive confidence intervals across the experiment's lifetime, each with its own significance value. The numbers you see on the Results page are deliberately conservative summaries of that series:

The statistical significance shown is the smallest significance value observed across those sequential intervals, not the average and not the latest.
The confidence interval shown is the running intersection of all prior intervals: it tracks the smallest upper limit and the largest lower limit seen during the run.

Because of this, the displayed significance and interval may not exactly match the currently observed conversion rates. That is intentional. It is what makes the result robust to having been observed many times.

Why Significance Climbs (and Occasionally Drops)

In a stable environment, significance should rise in a stepwise, generally increasing fashion as evidence accumulates. Two forms of evidence move it: larger differences between conversion rates, and differences that persist across more visitors. Early on, when the sample is small, large swings are treated conservatively, so you often see a flat line that later rises sharply once real evidence accrues.

Significance can also fall, though Optimizely's analysis indicates this happens in only about 4% of experiments. Small dips of a few percentage points come from time bucketing: Optimizely divides the experiment's runtime into 100 equal buckets that grow as the test runs, reshuffling visitors among them and recomputing as it goes, which produces minor fluctuations. Larger drops, potentially all the way to 0%, come from a stats reset, a protective mechanism that triggers when the engine detects that the underlying environment has changed (the assumption that data is identically distributed has been violated). A reset is the engine refusing to stand behind a conclusion the new evidence no longer supports.

False Discovery Rate Control vs Traditional Significance

The second pillar is what makes Stats Engine trustworthy when you run many metrics and variations. The naive error metric is the false positive rate: out of all the comparisons where there is truly no effect, what fraction are wrongly flagged? Optimizely controls something more decision-relevant, the false discovery rate: out of the results you would actually act on (the declared winners and losers), what fraction are wrong?

The distinction matters enormously. Optimizely's worked example: an experiment with ten comparison opportunities reports two winners, one of which is a false winner. Measured as a false positive rate, that is 1 in 10, about 10%, which sounds acceptable. But you do not implement the eight inconclusive results; you implement the two winners. Among those, your error rate is 1 in 2, or 50%. The false discovery rate captures the risk that actually reaches your roadmap.

False positive rate = false positives / all null comparisons   = 1/10 = 10%
False discovery rate = false positives / declared discoveries   = 1/2  = 50%

To control FDR across many hypotheses, Stats Engine uses a tiered version of the Benjamini-Hochberg procedure. The tiering reflects that not all metrics deserve equal weight:

Primary metric (rank 1) — evaluated independently of all others, so it reaches significance as fast as possible and is unaffected by how many other metrics you track.
Secondary metrics (ranks 2-5) — their significance threshold is adjusted for the number of metrics and variations. Adding more secondary metrics can slow each of them to significance, but never slows the primary metric.
Monitoring metrics (rank 6+) — each given a fractional weight of 1/n, so they have minimal impact on secondary metrics and none on the primary metric.

The practical payoff: Optimizely keeps the false discovery rate low (approximately 10%) while still letting your most important metric reach significance quickly. This is also why Stats Engine uses two-tailed tests, which are required for FDR control.

One caveat the documentation is explicit about: FDR control is not maintained when you segment results. The deeper you slice, the higher your chance of finding a spurious "significant" segment. Use segments for exploration, not for decisions, and only inspect the most meaningful ones.

How to Read Significance and Confidence Intervals

Statistical significance answers a precise question: how unusual would these results be if the variation and baseline truly performed identically? At 90% significance, you are accepting roughly a 10% false-positive risk on that call. The confidence interval is the estimated range that likely contains the true effect (the true uplift), and Optimizely sets its confidence level to match your project's significance threshold (90% by default).

The single most useful rule for reading the Results page:

A variation reaches significance exactly when its confidence interval stops crossing zero.

Confidence interval entirely above 0% means a winning variation.
Confidence interval includes 0% means inconclusive (you cannot yet rule out "no effect").
Confidence interval entirely below 0% means a losing variation.

Before declaring anything, Stats Engine enforces minimum data thresholds. For binary metrics, it requires at least 100 visitors or sessions and at least 25 conversions in both the baseline and the variation. For numeric metrics such as revenue, it requires at least 100 visitors or sessions but no fixed conversion count. Until those are met, the page reports that more visitors are needed and estimates the wait.

A useful judgment heuristic from the documentation: if the observed mean (the tick mark) sits near the edge of the confidence interval, the engine may be accumulating evidence against its own conclusion, so consider waiting. If the observed mean sits near the center, you can be more confident the call will hold.

For revenue-per-visitor goals, be aware that revenue distributions are heavily skewed, which reduces statistical power. Stats Engine applies skew correction to recover some of that power and to shape the confidence interval correctly, but skewed metrics still generally need more data than binary ones.

Stats Accelerator: A Separate Feature, Not the Engine

Stats Accelerator is frequently conflated with Stats Engine. They are different things. Stats Engine is the statistical methodology that evaluates results. Stats Accelerator is a traffic-allocation feature that sits on top of it and uses a multi-armed-bandit-family algorithm (a variation on the Upper Confidence Bound strategy) to shorten the time to statistical significance.

Stats Accelerator monitors a running experiment and routes more traffic toward the variation showing the most significant difference from the baseline, regardless of whether that difference is positive or negative, because its goal is to minimize time, not regret. Once a variation reaches significance, it is removed from consideration and traffic is redistributed to the rest. It requires at least three variations (a baseline plus two). It still produces statistical significance, because the underlying engine is still doing the inference.

This is distinct from a true multi-armed bandit** (MAB) optimization** (formerly "Accelerate Impact"), which minimizes regret by funneling traffic to whichever variation currently performs best on the primary metric. MABs are for temporary, value-maximizing scenarios such as a Black Friday promotion, and crucially MAB optimizations do not generate statistical significance at all.

Feature	Goal	Produces significance?	Use when
Stats Accelerator	Minimize time to significance	Yes	You want a reliable winner faster
Multi-armed bandit	Maximize reward / minimize regret	No	Short-lived campaigns; no permanent decision needed

Because Stats Accelerator changes traffic allocation mid-flight, it risks a sampling bias called Simpson's Paradox when conversion rates vary over time (for example, weekday-vs-weekend seasonality). Optimizely addresses this with the Epoch Stats Engine, which produces a stratified, weighted improvement estimate, comparing baseline and variation within each interval between allocation changes, then combining those intervals by visitor count. This is also why, with Stats Accelerator enabled, the Results page may report both absolute (in percentage points) and relative improvement. For Feature Experimentation, use a user profile service (sticky bucketing) so frequent reallocation does not expose one visitor to multiple variations.

Common Misconceptions

Is the Optimizely Stats Engine Bayesian?

No. This is the most common misconception, and it is worth correcting precisely. Stats Engine is a frequentist sequential method. Optimizely does offer a separate, explicitly Bayesian A/B testing mode (which expresses results as direct probabilities like "90% chance B beats A"), and a separate Frequentist Fixed Horizon mode. But the classic Stats Engine, the one that powers sequential testing, is frequentist. It reports statistical significance and frequentist confidence intervals, not posterior probabilities. The reason a search for "Optimizely Bayesian" surfaces Stats Engine at all is that both Bayesian and sequential methods let you peek and stop early. That shared behavior does not make them the same methodology.

Stats Engine results disagree with my t-test, so they must be wrong

They can legitimately disagree, and Stats Engine is the more trustworthy of the two when you have been monitoring continuously. A t-test uses only the currently observed mean and difference, so if evidence looked strong early and weakened later, a t-test can still report the stale, strong result. Stats Engine's intersected intervals are more conservative, less likely to declare a false winner, and less likely to reverse a call later.

I can keep slicing segments until something is significant

You can, but you will be manufacturing false discoveries. FDR control does not extend across segments. Repeated segment-hunting inflates false positives exactly like peeking does.

A stats reset means the tool is broken

The opposite. A reset means the engine detected that the environment changed and is protecting you from standing behind a conclusion the new data no longer supports.

Practical Guidance for Trusting Your Results

Rank your metrics deliberately. Put the metric that defines success as the primary metric, ideally measured close to the change in the funnel. It gets independent, fastest-to-significance treatment; everything else is secondary or monitoring.
Let the test run to its planned duration even though you can peek. Sequential validity means peeking will not break your stats, but a result that has barely cleared the threshold on thin data is fragile. Treat experimentation as a standardized process, not a dashboard you babysit.
Read the confidence interval, not just the significance number. Width tells you precision; position relative to zero tells you direction; the tick mark's position warns you whether a call is at risk of reversing.
Use segments to explore, never to decide. If a segment looks interesting, treat it as a hypothesis for a new, properly powered experiment.
Match the method to the intent. Use a standard A/B test (sequential Stats Engine) when you need a trustworthy decision; add Stats Accelerator to reach that decision faster; use a multi-armed bandit only for temporary value-maximization where you do not need a statistically defensible winner.

For how the three inference families differ in practice, see Bayesian vs frequentist vs sequential testing.

Related guides

Feature Flag Best Practices for Production Systems

David Sert — Thu, 20 Aug 2026 05:07:11 +0000

Your flag count has quietly crossed a threshold. What started as a clean way to ship a risky feature behind a toggle is now a few hundred conditionals scattered across services, half of them permanently on, nobody quite sure which are safe to delete. Someone flips the wrong one during an incident. A test passes locally and fails in CI because a flag defaulted differently. This is the predictable midlife of feature flagging: the mechanism is trivial, but the discipline around it is what separates teams who ship faster from teams who have simply moved their risk somewhere less visible.

The good news is that the disciplines are well understood and largely independent of any one vendor. This article lays them out, then shows how they map onto Optimizely Feature Experimentation specifically, so that the practices have a concrete home in tooling rather than living only in a wiki page nobody reads.

Classify every flag by type and lifespan

The single most useful idea in feature flagging is that not all flags are the same thing. Martin Fowler's taxonomy splits them into four categories, and the reason it matters is that each has a different owner, a different lifespan, and a different failure mode when you ignore it (Fowler, Feature Toggles).

Release toggles hide in-progress work so it can merge to trunk and ship dormant. They are the shortest-lived flags you own, meant to survive days or weeks, and they should be deleted the moment the feature is fully rolled out.
Experiment toggles split traffic between variations to measure an outcome. They live exactly as long as the test needs to reach a decision, then collapse to the winning variation. This is the layer beneath A/B testing.
Ops toggles give operators control over system behavior: degrade an expensive feature under load, disable a flaky dependency, or trip a kill switch during an incident. Some are short-lived, but a genuine kill switch may live indefinitely as a safety control.
Permissioning toggles decide who gets access to what, such as gating a feature to premium accounts. These can legitimately live for years and are effectively part of your product, not temporary scaffolding.

Fowler frames these along two axes: longevity (how long the flag should exist) and dynamism (whether the decision is fixed per deployment or must be re-evaluated per request). The categories fall at different points, and conflating them is the root of most flag debt. A release toggle that gets treated like a permanent ops switch never gets cleaned up, because nobody remembers it was supposed to be temporary.

The practical takeaway is to make the type explicit at creation. Decide, before you write the flag, whether it is a release, experiment, ops, or permissioning flag, and record that alongside an owner and an expected removal or review date. In Optimizely, flag rules make the distinction concrete: an A/B test rule is an experiment flag, while a targeted delivery is a release or ops flag. If you are weighing a fixed split against an adaptive one for an experiment flag, the trade-offs are covered in Stats Accelerator vs multi-armed bandit vs contextual bandit.

flowchart TD
    A[New flag] --> B{What is its job?}
    B -->|Hide in-progress work| C[Release toggle<br/>lifespan: days-weeks<br/>delete after rollout]
    B -->|Split traffic to measure| D[Experiment toggle<br/>lifespan: the test<br/>collapse to winner]
    B -->|Operator control / kill switch| E[Ops toggle<br/>lifespan: short, or<br/>indefinite for safety]
    B -->|Gate access by plan/role| F[Permissioning toggle<br/>lifespan: years<br/>part of the product]
    C --> G[Book its removal now]
    D --> G
    E --> H[Owner + review date]
    F --> H

Name flags so they stay searchable at scale

At ten flags, naming is a matter of taste. At three hundred, it is the difference between a flag you can find and a flag you are afraid to touch. A good flag key is greppable across your codebase, self-describing, and encodes enough context that someone who did not create it can guess its purpose and owner.

A convention worth adopting encodes, in a consistent order, the owning team or domain, the flag's intent, and where it applies. Something like checkout_new-address-form_release or search_semantic-ranking_experiment reads cleanly, sorts sensibly, and tells a reader the type without opening a dashboard. Keep to one delimiter, avoid spaces and punctuation, and never reuse a retired key for a new purpose. OptiPilot's free feature flag naming convention generator turns a flag's type, team, and feature into a consistent key across dot, kebab, and snake styles, so a whole team names flags the same way.

Naming discipline pays off precisely because flag keys are the join between your code and your configuration. Optimizely makes this contract literal: a rule's key cannot be changed after creation, and if the key referenced in your decide call does not match a key in the platform, no traffic is served to that rule (Manage rules). A typo is not a loud error; it is a silent no-op that quietly serves the fallback. Consistent, reviewed names make that class of mistake far easier to catch in code review.

Roll out gradually and keep a kill switch

The reason flags earn their keep in production is that they decouple deploy from release. You ship code dark, then turn it on for a slice of traffic, watch, and widen the slice, or reverse instantly if something breaks. Two properties make this safe: a gradual ramp and a fast reversal.

A gradual ramp means starting small. Optimizely's own guidance for feature rollouts recommends 10% as a starting allocation before widening (Run Feature Rollouts). You can also target the ramp: deliver to internal users first, then beta customers, then the general population, using audience conditions rather than a blunt percentage. Optimizely's targeted delivery supports exactly this, letting you roll out to a percentage of a specific audience and increase it over time (Flag delivery overview).

Fast reversal is the kill switch. Because flag changes take effect through configuration rather than a code deploy, you can roll a feature back in seconds. In Optimizely, deliveries take effect without a code deployment, and you can toggle a flag on and off without re-bucketing the users already assigned, as long as you do not change the traffic allocation at the same moment (Run flag deliveries). That last caveat matters operationally: an incident kill switch should flip the flag off, not simultaneously re-slice traffic, or you will disturb bucketing for everyone else.

There is a subtle trap in ramping that shows up most in experiment flags. Bucketing is deterministic: Optimizely hashes the user ID to assign a variation, so the same user gets the same experience as long as the distribution has not changed (Core concepts). Change the traffic allocation on a running experiment and users can be re-bucketed, which both disturbs their experience and can skew results. If you need to change allocation mid-flight, use a user profile service to keep bucketing sticky, and watch for sample ratio mismatch, the canary that tells you your split is not landing the way you configured it.

One decision to make deliberately: whether a given flag needs measurement. Optimizely separates a plain targeted delivery (free, no analytics, dispatches no decision events) from a feature rollout (full analytics, consumes monthly active users and impressions). Use a targeted delivery for a simple release or kill switch where you only need on/off control, and a rollout or A/B test when you actually want to measure impact. Reaching for the analytics-bearing rule on a flag you will never analyze is a common and needless cost.

Environments and the datafile: where flags actually live

Flags are configuration, and configuration needs somewhere to live per stage of your pipeline. In Optimizely, a project contains one or more environments, typically staging and production, and each environment holds its own ruleset for a flag. You can run a flag at 100% in development and 10% in production, or configure entirely different rules per environment, with separate permissions for who can change each (Core concepts).

At runtime the SDK does not call back to Optimizely on every decision. It downloads a datafile, a JSON snapshot of your flags and rules for one environment, and evaluates decisions locally against it. When you edit a rule, the datafile updates within a few seconds, and how quickly that reaches production depends on how often your app fetches it; webhooks let you push updates in near real time (Manage rules). This architecture is why flag evaluation is fast and why it keeps working during a brief network blip: decisions are computed from an in-memory config, not a synchronous network round trip.

It also defines your failure mode, which is the next thing to get right.

Test safely with flags and sensible fallbacks

Flags multiply the states your system can be in, and untended they make tests flaky and behavior unpredictable. Three habits keep that under control.

First, always have a safe default. Design so that the flag being off, or the configuration being unavailable, yields a working, conservative experience. Optimizely's decide method returns an OptimizelyDecision object whose enabled field is false until a rule turns the flag on, and on a critical error, such as the SDK not being ready or an unknown flag key, it returns a nullvariationKey and populates reasons (Decide methods). Your code should treat that state as "off" and fall back to a value you control, rather than assuming a variable will be present. For resilience against the SDK not having a datafile yet, you can also bundle a datafile with the application so the client can initialize synchronously from a known-good snapshot.

Second, decouple the decision from the decision point. Fowler's advice is to avoid scattering raw flag checks throughout the codebase, because changing the logic then means hunting down every call site. Instead, funnel decisions through a single abstraction, a FeatureDecisions object or equivalent, so the toggle point (where behavior branches) is separated from the toggle router (how the decision is made). This keeps business logic testable without the flagging system present, and confines flag removal to one place.

Third, test the states you actually ship. At minimum, exercise the flag-on and flag-off paths, since both are real production states. Wrap decisions so tests can inject a decision directly rather than standing up the SDK, which removes network dependence and the flakiness that comes with it. For the human side of pre-launch verification, a structured pass like the experiment QA checklist catches the targeting, bucketing, and event-tracking mistakes that unit tests miss.

Pay down flag debt before it compounds

Every flag is inventory, and inventory has a carrying cost. Fowler's framing is blunt: "savvy teams view their feature toggles in their codebase as inventory which comes with a carrying cost and seek to keep that inventory as low as possible." Each live flag is an extra branch to reason about, an extra combination to test, and an extra thing that can be misconfigured. Stale flags are the single most common feature-flag problem, and they accumulate because deleting a flag is nobody's urgent job.

The fix is to make cleanup a scheduled, owned activity rather than a good intention:

Book the removal when you create the flag. For every release toggle, add a removal task to the backlog at the moment it is introduced. The work of retiring it is then already tracked, not rediscovered months later.
Give short-lived flags an expiry. Record an expected removal date on release and experiment flags. Some teams go further and build "time bombs" that fail a test, or even refuse to start the application, once a flag outlives its expiration date, converting silent debt into a loud, actionable failure.
Cap the inventory. A Lean-style limit on the total number of live flags forces a retirement before a new flag can be added, keeping the pile bounded.
Review the whole set on a cadence. Optimizely's flags dashboard gives you the raw material: every flag with its status, creator, and last-modified date, filterable by environment and audience (Custom Flags Dashboard). A monthly sweep for flags that are fully rolled out, concluded, or untouched for a quarter turns "someone should clean these up" into a concrete list.

When a flag is genuinely done, retire it in both places. Remove the code branch and the flag definition together, so you never leave a dangling key or an orphaned conditional. The code side of that sweep can be automated: find unused feature flags walks through locating stale flag references and dangling keys in your codebase before they calcify. In Optimizely, archiving a flag removes it from the datafile and requires that it not be active in your highest-priority environment, usually production, which is a useful guardrail against archiving something still serving traffic; the flag's data is retained so you can unarchive later if needed (Custom Flags Dashboard).

How Optimizely maps flags to your code

Pulling the pieces together, here is the shape of the integration. A flag is, in Optimizely's own words, "a place in code where a decision is made." You create a user context from a user ID and any targeting attributes, then call decide with the flag key. The returned decision tells you whether the flag is enabled and carries any configuration variables and the variation key (Key concepts).

The following is illustrative JavaScript, showing the decision routed through a single wrapper with an explicit fallback. Method names are verified against current SDK docs, but exact syntax varies by language and SDK major version.

// Illustrative: one place that owns the decision, with a safe default.
const FALLBACK_SORT = 'alphabetical'; // conservative, always-works behavior

async function getProductSort(optimizely, userId, attributes) {
  // Wait until the SDK has a datafile; on failure, fall back.
  try {
    await optimizely.onReady();
  } catch (err) {
    return { enabled: false, sortMethod: FALLBACK_SORT };
  }

  const user = optimizely.createUserContext(userId, attributes);
  const decision = user.decide('product_sort');

  // Critical error: variationKey is null and reasons are populated.
  // Treat as "off" and use our own fallback rather than trusting variables.
  if (decision.variationKey === null) {
    console.warn('flag decision failed:', decision.reasons);
    return { enabled: false, sortMethod: FALLBACK_SORT };
  }

  return {
    enabled: decision.enabled,
    sortMethod: decision.enabled
      ? decision.variables['sort_method']
      : FALLBACK_SORT,
  };
}

Two SDK details are worth knowing for production hygiene. First, decide dispatches a decision (impression) event by default so the exposure shows up in results; when you only want to read a flag's state without recording an exposure, the OptimizelyDecideOption.DISABLE_DECISION_EVENT option suppresses it (Decide methods). This is the right tool when you evaluate a flag early to pre-render content and will record the real exposure later at the point of interaction. Second, remember the silent-failure rule from earlier: a flag key in your code with no matching, running rule serves no traffic, so keep code and configuration in lockstep.

The through-line across all of this is modest but powerful. Treat each flag as a small, owned, dated commitment rather than a permanent fixture. Name it so it can be found, ramp it so a mistake is cheap, keep a switch that reverses it fast, and schedule its removal the day you create it. Do that consistently and feature flags go back to being what they were supposed to be: a mechanism for shipping with confidence, not a second codebase you are quietly maintaining by accident.

To see a flag decision run end to end, open the demo lab. The JavaScript v6+, React and Next.js SSR harnesses each make a live decision and show its events. Teams still on v5 can use the JavaScript Browser v5 harness.

False Discovery Rate Control in A/B Testing

David Sert — Thu, 20 Aug 2026 05:06:35 +0000

Track one metric on an A/B test and a false positive is a 1-in-20 event at the usual 95% confidence threshold. Track twenty metrics and, under the null hypothesis, you should expect one of them to look like a winner purely by chance. Add a few variations and a segment breakdown, and the number of simultaneous comparisons climbs into the dozens. This is the multiple comparisons problem, and left uncontrolled it turns a rich results page into a machine for manufacturing phantom wins. The false discovery rate is the modern statistical answer to it — a way to keep testing many things at once without letting your rate of false claims run away from you.

This guide explains what the false discovery rate is, how it differs from older corrections like family-wise error control and Bonferroni, how the Benjamini-Hochberg procedure works with a fully worked example, and how Optimizely's Stats Engine applies false discovery rate control automatically across every metric and variation on the results page.

What Is the False Discovery Rate?

The false discovery rate (FDR) is the expected proportion of your "discoveries" — the results you flag as statistically significant — that are actually false positives. If you declare 10 winning metrics and your FDR is controlled at 5%, then on average you expect about 0.5 of those 10 to be a mistake. FDR is a property of the set of significant results, not of any single test.

That framing is the whole point. A traditional p-value controls the error rate of one hypothesis in isolation: a 5% significance level means that if this one variation has no real effect, there is a 5% chance you wrongly call it significant. But an experiment rarely tests one hypothesis. It tests a primary metric, several secondaries, a stack of guardrails, maybe two or three variations, and often a few segments. Every one of those cells is a hypothesis test. The false discovery rate asks a more useful question: across everything I flagged as a discovery, what fraction should I expect to be wrong?

Controlling the FDR lets you keep a wide, informative results page — which you want, because catching harm requires watching many metrics — while bounding how often you chase something that was never really there.

The Multiple Comparisons Problem

The mechanism is pure arithmetic. Suppose every metric on your experiment is truly null (no real difference between variation and control), and you test each at a 5% significance level. The probability that a single test does not produce a false positive is 0.95. The probability that none of m independent tests produces one is 0.95^m, so the probability of at least one false positive is:

P(at least one false positive) = 1 - 0.95^m

m = 1   metric   -->  5.0%
m = 5   metrics  -->  22.6%
m = 10  metrics  -->  40.1%
m = 20  metrics  -->  64.2%
m = 50  metrics  -->  92.3%

At twenty metrics you are more likely than not to see at least one "significant" result even when nothing is happening. This is why a naive results page that runs every metric at a flat 95% threshold is untrustworthy the moment it carries more than a handful of metrics: the headline significance numbers no longer mean what they appear to mean. The problem compounds with variations and segments, because each comparison is another draw from the same lottery.

The instinct to fix this by simply "watching fewer metrics" is the wrong trade. You need guardrail metrics to catch changes that win on conversion but quietly raise refunds or slow the page — that is a deliberately wide net. The right fix is not to look at less, but to adjust the significance bar to account for how much you are looking at.

Family-Wise Error Rate vs False Discovery Rate

There are two different error rates you might try to control, and the choice shapes everything downstream.

The family-wise error rate (FWER) is the probability of making even one false positive across the entire family of tests. Controlling FWER at 5% means there is at most a 5% chance that any of your significant results is wrong. This is a strict, conservative standard — appropriate when a single false claim is catastrophic, such as a drug-approval trial.

The false discovery rate (FDR) is the expected proportion of false positives among the results you called significant. Controlling FDR at 5% means you accept that some individual discoveries may be wrong, as long as, on average, no more than 5% of them are.

	Family-wise error rate (FWER)	False discovery rate (FDR)
Controls	Probability of any false positive	Expected fraction of false positives among discoveries
Stance	One mistake is unacceptable	A bounded share of mistakes is tolerable
Power	Low — misses many real effects	High — detects far more real effects
Typical procedure	Bonferroni, Holm	Benjamini-Hochberg
Best fit	Small number of critical tests	Many exploratory tests, as in A/B testing

For online experimentation, FDR is almost always the better target. You run many tests, most effects are small, and the cost of missing a real winner (low power) is a genuine business cost. FDR control gives you dramatically more statistical power than FWER control while still keeping false claims on a leash. That trade — accept a known, bounded fraction of errors in exchange for detecting many more true effects — is why FDR has become the default for large-scale testing.

The Benjamini-Hochberg Procedure

The Benjamini-Hochberg (BH) procedure, introduced by Yoav Benjamini and Yosef Hochberg in 1995, is the standard method for controlling the false discovery rate. It is refreshingly mechanical.

Given m p-values and a target FDR level q (say 0.10):

Sort the p-values in ascending order, so p(1) <= p(2) <= ... <= p(m).
For each rank k, compute the BH critical value (k / m) * q.
Find the largest rank k for which p(k) <= (k / m) * q.
Declare significant every hypothesis from rank 1 up to and including that k — even any whose raw p-value sits above the naive threshold.

The key idea is the sloped threshold. Instead of comparing every p-value to one fixed cutoff, BH compares each ranked p-value to a line that rises with its rank. A p-value only needs to beat (k / m) * q, which is more forgiving for higher ranks when there are many small p-values supporting them. The procedure adapts: when the data contain many strong effects, the effective cutoff loosens; when almost nothing is real, it tightens toward the strict Bonferroni bound.

flowchart TD
    A[Collect m p-values from every metric and variation] --> B[Sort ascending: p1 to pm]
    B --> C[For each rank k compute BH critical value k over m times q]
    C --> D{Find largest k where p_k is at or below its BH critical value}
    D --> E[Reject ranks 1 through k: declare significant]
    D --> F[Ranks above k: not significant]
    E --> G[Expected false positives among rejections stays at or below q]

A Worked Example

Take 10 metrics on an experiment and a target FDR of q = 0.10. Sort the p-values ascending, compute each BH critical value (k / 10) * 0.10 = k * 0.01, and compare:

Rank k   p-value    BH critical (k/m)*q    p <= critical?
  1      0.001         0.010                yes
  2      0.008         0.020                yes
  3      0.012         0.030                yes
  4      0.021         0.040                yes
  5      0.030         0.050                yes
  6      0.041         0.060                yes
  7      0.060         0.070                yes   <-- largest passing rank
  8      0.200         0.080                no
  9      0.350         0.090                no
 10      0.600         0.100                no

Largest k with p(k) <= (k/m)*q  is  k = 7.
Benjamini-Hochberg rejects ranks 1 through 7  ->  7 discoveries.

Notice rank 7: its p-value is 0.060, which is above the naive 0.05 threshold, yet BH still declares it significant. That is the procedure working as designed — the weight of six even-smaller p-values below it earns rank 7 a place. Now compare the same data under a strict Bonferroni correction, which would test every metric against 0.10 / 10 = 0.01: only ranks 1 (0.001) and 2 (0.008) clear that bar, for 2 discoveries instead of 7. Same data, same 10% error budget — BH finds more than three times as many real effects because it spends that budget across the whole set instead of on each test in isolation.

FDR Control vs Bonferroni Correction

The Bonferroni correction is the best-known multiple-comparisons fix: divide your significance level by the number of tests, so with 20 metrics at 5% each metric must clear 0.05 / 20 = 0.0025. It controls the family-wise error rate, and it is trivially simple. It is also badly suited to A/B testing.

	Bonferroni	Benjamini-Hochberg (FDR)
Controls	Family-wise error rate	False discovery rate
Threshold	Fixed: `alpha / m` for every test	Sloped: `(k / m) * q`, rank-dependent
As metrics grow	Cutoff shrinks fast; power collapses	Cutoff adapts; power preserved
False negatives	Many — misses real winners	Far fewer
Right context	A few make-or-break tests	Many metrics, exploratory testing

Bonferroni's flaw in this setting is that it treats every additional metric as equally threatening and pays for it with a cutoff that shrinks toward zero. With 50 metrics, each must beat 0.001 — so real, moderate effects get thrown out alongside the noise, and your false-negative rate soars. FDR control accepts a small, known fraction of false positives in exchange for keeping the power to detect the effects that matter. For an experimentation program that lives or dies by finding true wins across a broad metric set, that is the correct trade.

How Optimizely's Stats Engine Controls the False Discovery Rate

Optimizely's Stats Engine applies false discovery rate control automatically. Per Optimizely's documentation, the Stats Engine "provides a data-rich view of visitor interactions, includes confidence intervals, and applies false discovery rate control" across the metrics on the Experiment Results page. You do not configure it, sort p-values, or apply a correction by hand — it is built into how significance is reported. For the broader mechanics of how significance and always-valid inference are computed, see How the Optimizely Stats Engine Works.

Two documented properties are worth internalizing:

Testing more does not raise your false-positive risk. Optimizely states that the "Stats Engine adjusts statistical significance for every metric and variation, so testing more does not raise the risk of false positives." That is the FDR guarantee in plain language — the correction scales with how many comparisons the results page carries, so adding a guardrail does not silently inflate your chance of a phantom winner.
The primary metric is protected. On the results page, "the primary metric's statistical significance is calculated independently from all other metrics and monitoring goals." The tiering matters: your headline result reaches significance as fast as possible, while secondary and monitoring metrics absorb the multiple-comparisons adjustment. Adding many low-improvement secondary metrics slows their time to significance, but never the primary's.

This is why the Change the statistical significance setting documentation can promise that carrying a wide metric set — including a full complement of guardrail metrics — does not cost you in false alarms. FDR control is what makes a broad, honest results page statistically safe.

Do Not Change Metrics Mid-Experiment

There is one operational rule that follows directly from how FDR control is computed. Optimizely controls the false discovery rate over "metrics currently appearing on the Experiment Results page." Its Edit a metric documentation warns that changing the metrics during a running experiment "alters the number and nature of the hypotheses being tested, which can affect the false discovery rate control calculations," and that "excessive metric changes after an experiment or campaign has started may invalidate Optimizely's guarantees regarding false detection."

The reason is exactly the BH mechanism above: the critical values depend on m, the number of hypotheses. Add or swap metrics after data has started arriving and you retroactively change m, which shifts every threshold and reopens the door to the cherry-picking FDR control was meant to close. Decide your metric set before launch and freeze it — the same discipline that protects you from the winner's curse and from sequential testing peeking traps.

Practical Implications for Your Experiments

Understanding FDR control changes a few concrete habits:

Keep the metrics you need, not fewer. Because the Stats Engine controls FDR across the results page, you can carry a healthy set of guardrails and secondaries without inflating false positives. Trim metrics for clarity, not out of statistical fear.
Lock the metric set at launch. Write your primary, secondary, and guardrail metrics into the experiment brief before traffic starts, and do not edit them mid-flight. Changing m mid-experiment undermines the FDR guarantee.
Read the primary metric first. It is computed independently and reaches significance fastest. Treat secondary and monitoring metrics as supporting evidence whose significance the FDR adjustment deliberately makes more conservative.
Power the experiment for the metrics that matter. FDR control makes false positives rare, but it cannot rescue an underpowered test from false negatives. Size the experiment properly — see A/B test sample size and statistical power — so you can actually detect the effects you care about.
Validate the traffic split. No error-rate control matters if the experiment is broken at the instrumentation level. Check for sample ratio mismatch before you trust any significance number on the page.

Frequently Asked Questions

What is a good false discovery rate to target?

For most online experimentation, controlling the FDR at 5% to 10% is standard — the same range as conventional significance levels. A 10% FDR means you accept that, on average, up to one in ten of your flagged discoveries may be a false positive, in exchange for substantially more power to detect real effects. Optimizely ties this to your project-wide statistical significance setting rather than asking you to specify an FDR directly.

Is the false discovery rate the same as the false positive rate?

No, and the distinction is important. The false positive rate (or Type I error rate) is the probability of a false positive among all truly null tests — it looks backward from the ground truth. The false discovery rate is the expected fraction of false positives among the results you declared significant — it looks forward from your discoveries. FDR is the more actionable quantity because it describes the reliability of the conclusions you actually act on.

Why not just use Bonferroni correction?

Bonferroni controls the family-wise error rate by dividing your significance level across all tests, which makes the threshold shrink rapidly as you add metrics. With many metrics it becomes so conservative that it discards real effects along with the noise, driving up false negatives. Benjamini-Hochberg controls the false discovery rate instead, adapting the threshold to the data and preserving far more power — a better fit for the many-metric reality of A/B testing.

Does tracking more metrics in Optimizely increase my false positive risk?

No. Optimizely's documentation states that the Stats Engine "adjusts statistical significance for every metric and variation, so testing more does not raise the risk of false positives." The false discovery rate control scales with the number of comparisons on the results page. Adding metrics can lengthen the time for secondary metrics to reach significance, but it does not inflate your overall false-positive risk.

Why does changing metrics mid-experiment break FDR control?

Benjamini-Hochberg critical values depend on the number of hypotheses being tested. When you add, remove, or edit metrics on a running experiment, you change that number retroactively, which shifts the thresholds the Stats Engine already applied and can invalidate Optimizely's guarantees against false detection. Decide your metric set before launch and keep it fixed for the life of the experiment.

Experiment Design for A/B Testing: A Practical Framework

David Sert — Thu, 20 Aug 2026 05:05:59 +0000

An A/B test can be implemented perfectly and still answer the wrong question. Experiment design is the written contract that connects a business decision to a causal comparison: who is eligible, what is randomized, what changes, how outcomes are measured, and what evidence will trigger action.

This guide turns those choices into a one-page specification, then maps the same design to Optimizely Web Experimentation and Feature Experimentation. It complements the deeper guides to an A/B testing hypothesis, experiment metrics, sample size and statistical power, test duration, and pre-launch QA.

Experiment design is the blueprint for a causal answer

The output of experiment design is not a list of variation ideas. It is a precise description of the comparison that will support a decision. Optimizely's own basic experiment plan asks why the experiment runs, where and when variations run, who sees them, what changes, and how success is measured. A rigorous design adds the unit of randomization, estimand, power assumptions, and decision rule.

flowchart LR
  A[Business decision] --> B[Causal question]
  B --> C[Unit and eligibility]
  C --> D[Control and treatment]
  D --> E[Metrics and estimand]
  E --> F[MDE, sample, and duration]
  F --> G[Optimizely configuration]
  G --> H[QA and launch]
  H --> I[Ship, iterate, stop, or conclude]

Start with the decision the result must support

Write the action before the hypothesis. A useful decision is specific enough that a result can change it:

Ship the checkout recommendation module to all eligible customers if it produces a practically valuable increase in completed orders without materially increasing refund rate; otherwise iterate or stop.

Turn that decision into one falsifiable causal question:

Among eligible signed-in shoppers, what is the effect of showing the recommendation module during checkout, compared with the current checkout, on completed orders within seven days of assignment?

This order prevents a common failure: designing an attractive variation, choosing an available metric, and only later deciding what the test was meant to prove.

Define the estimand before the dashboard metric

The estimand is the effect the team wants to estimate. Define five elements:

Element	Checkout recommendation example
Population	Signed-in shoppers who begin checkout in supported markets
Treatment	Checkout with the recommendation module enabled
Comparator	Current production checkout with the module absent
Outcome	Orders completed within seven days of assignment
Time window	Seven days from the first eligible checkout

Use an intention-to-treat estimand by default: analyze people according to the variation to which they were assigned, even if rendering fails or they do not engage with the module. This preserves the benefit of random assignment and estimates the effect of the policy as deployed.

An exposure-only analysis answers a different question because it conditions on behavior or technical events that may happen after assignment. It can be useful as a diagnostic, but it should not silently replace the primary analysis.

Know when an experiment is the wrong tool

Do not randomize merely because the platform makes it possible. An experiment is a poor choice when:

No plausible outcome would change the decision.
Available traffic cannot detect an effect worth acting on within a useful timeframe.
The treatment cannot be assigned independently enough to create a credible control.
Random assignment would be unethical, unlawful, or operationally unacceptable.
Instrumentation cannot distinguish assignment, exposure, and outcomes reliably.

When assignment cannot be randomized but a decision still needs evidence, use a quasi-experimental design and state its assumptions explicitly. That design should not be presented as equivalent to a randomized test.

Choose the experimental unit and assignment mechanism

The experimental unit is the entity independently assigned to control or treatment. It determines what must remain stable, where interference can occur, and which observations are independent in the analysis.

User, session, account, device, or cluster

Match the unit to how the treatment is delivered and how people can influence one another:

Unit	Appropriate when	Main risk
User	A durable identity receives an individual experience	The same person appears under multiple IDs
Session	The treatment is intentionally temporary and session-contained	Repeat sessions receive inconsistent experiences
Account	Members share settings, workflows, or outcomes	Too few accounts for adequate power
Device	The experience is device-specific and identity is unavailable	Cross-device contamination
Cluster	Locations, teams, or networks can affect one another	Correlated outcomes reduce effective sample size

For a B2B account-level recommendation feature, randomizing individual members can contaminate the comparison: one member may configure the experience for colleagues, or team members may discuss it. Assign the account, then analyze with methods that respect account-level assignment.

Keep assignment stable

Use a durable identifier for the unit of randomization. If the treatment must persist across browser sessions and services, a transient session ID is not sufficient. Document how anonymous identities merge with authenticated identities and what happens when a user changes devices.

Optimizely Web Experimentation uses deterministic bucketing: it hashes a user ID or cookie with MurmurHash, so the same identifier maps to the same variation while the relevant configuration remains stable. Optimizely also warns that some traffic changes can rebucket visitors; the official bucketing guide recommends avoiding allocation changes after results accumulate.

The randomization unit is not necessarily the metric denominator. A user-randomized experiment may measure revenue per eligible user, orders per user, or average order value among orders. Each denominator answers a different question and must be fixed in the measurement plan.

Detect spillovers and network effects

Random assignment does not prevent treated units from changing control units' outcomes. Before launch, ask:

Can users communicate about or share the treatment?
Does treatment change inventory, marketplace liquidity, queue time, or prices for everyone?
Can one account member alter settings seen by other members?
Does a recommender learn from behavior pooled across variations?

If interference is plausible, assign clusters that contain the spillover, use a switchback or marketplace-specific design, or redefine the estimand. Record the remaining interference risk rather than assuming it away.

Define control, treatment, and eligibility precisely

A reproducible comparison needs versioned experiences and frozen entry rules. “Business as usual” and “all users” are not adequate specifications.

Make the control a real production baseline

Describe the control as thoroughly as the treatment:

User interface and copy.
Code path and service dependencies.
Default feature-flag values.
Existing personalization or recommendation logic.
Analytics events expected from the baseline.

For the checkout example, the control is “the current checkout with no recommendation component and no recommendation API call,” not simply “original.” This matters if production changes while the experiment is being prepared.

Change one causal idea at a time

One hypothesis can require several implementation edits. Adding a recommendation module may change layout, make an API request, add loading behavior, and introduce new tracking. Those edits express one causal idea if the intended decision is whether to deploy the module as a package.

If the decision is which component causes the effect—placement, copy, image, or algorithm—use separate experiments or a multivariate test. A bundled treatment cannot identify the effect of each ingredient.

Freeze inclusion and exclusion rules

Define eligibility before anyone inspects results:

Target pages, routes, or flag call sites.
Geography, language, device, browser, and app version.
Account tier, authentication state, and product entitlement.
Prior exposure and concurrent-experiment exclusions.
Employee, bot, fraud, and QA-traffic handling.
The exact moment a unit enters the experiment.

Changing eligibility after viewing outcomes creates a new analysis population. If a safety issue forces a change, record the time and configuration, then decide whether to restart rather than blending incompatible cohorts.

Design the measurement plan

Metrics operationalize the estimand. They should be specified as formulas and event rules, not only selected by name in a dashboard.

Choose one primary metric

The primary metric must be affected by the treatment, interpretable by stakeholders, and tied to the ship decision. Define:

Numerator and denominator.
Aggregation unit.
Attribution or conversion window.
Event source and deduplication key.
Direction of improvement.
Practical effect threshold.

For the checkout example, “conversion” is ambiguous. A better definition is “distinct eligible users with at least one completed order within seven days divided by distinct assigned eligible users.”

Optimizely Web Experimentation treats the first metric added as the primary metric; its experiment setup guide recommends choosing an event directly affected by the change. Put the decision metric first deliberately.

Add restrained secondary and guardrail metrics

Secondary metrics explain mechanism or downstream impact. For the recommendation module, useful secondary measures might include recommendation clicks, units per order, and revenue per eligible user.

Guardrails describe harms that can veto a nominal win: refund rate, checkout latency, payment errors, support contacts, or gross margin. State each veto rule in advance. A dashboard containing dozens of metrics invites selective interpretation; include only metrics with a defined role.

Instrument assignment and outcomes separately

An outcome event is not proof that assignment or exposure worked. Validate three layers:

Assignment: the unit was bucketed into a variation.
Exposure: the intended experience was actually delivered.
Outcome: the conversion or guardrail event occurred and was attributed correctly.

Specify how the pipeline handles duplicate events, bots, missing identifiers, delayed events, identity merges, and events arriving after the experiment stops. Run an A/A or staged validation when the cost of a broken decision pipeline is high.

Design for enough information, not merely significance

A design should collect enough information to make the business decision. Statistical significance alone does not establish that an effect is large enough, safe enough, or measured without bias.

Pick an MDE worth acting on

The minimum detectable effect (MDE) should reflect the smallest effect that would justify implementation and opportunity cost. Derive it from baseline performance, expected value, engineering cost, risk, and competing roadmap items.

Do not inflate the MDE simply to make the required sample fit available traffic. That creates a faster test by declaring smaller—but potentially valuable—effects invisible.

Calculate sample size and calendar duration

Use the baseline rate, MDE, significance threshold, desired power, and number of variations to estimate the required sample. Then convert eligible sample into calendar time using:

The share of traffic satisfying eligibility.
Total experiment allocation.
Variation traffic distribution.
Weekly and seasonal cycles.
Ramp time and planned outages.
Conversion delay and data-processing latency.

The sample-size guide covers the calculation; the design document should retain the inputs and result. The duration guide explains why reaching a nominal sample early does not erase calendar-cycle or delayed-conversion requirements.

Predefine the stopping and decision rules

Write the complete rule before launch:

Statistical method and confidence policy.
Minimum runtime and information requirement.
Practical-effect threshold or acceptable interval.
Guardrail vetoes.
Safety-stop conditions.
Treatment of inconclusive results.

Do not stop because a preferred variation briefly looks significant, change the primary metric after seeing results, or segment repeatedly until a favorable result appears. Optimizely Stats Engine supports ongoing results monitoring, but operational visibility is not permission to rewrite the design after outcomes are known.

Map the experiment design into Optimizely

The same causal design can be implemented in either Web Experimentation or Feature Experimentation. The platform configuration enforces parts of the design; it does not choose the question, unit, estimand, MDE, or decision threshold for the team.

Start from one design card

Use a one-page design card as the source of truth:

Field	Checkout recommendation design
Decision	Ship, iterate, or stop the module
Causal question	Effect of offering recommendations on seven-day order completion
Population	Signed-in checkout starters in supported markets
Randomization unit	Durable user ID
Control	Current checkout without recommendation call or component
Treatment	Checkout with recommendation module
Primary metric	Seven-day completed-order users per assigned user
Secondary metrics	Recommendation clicks, units per order, revenue per user
Guardrails	Refund rate, checkout errors, p95 latency
Allocation	100% of eligible traffic
Distribution	50% control, 50% treatment
Runtime	Precomputed minimum sample plus complete weekly cycles
Decision rule	Practical lift and statistical evidence with no guardrail veto

flowchart TB
  D[Shared experiment design card]
  D --> W1[Web: URL or saved page]
  D --> W2[Web: audience]
  D --> W3[Web: original and variation]
  D --> W4[Web: allocation, distribution, metrics]
  D --> F1[Feature: flag and variables]
  D --> F2[Feature: attributes and audience]
  D --> F3[Feature: A/B test rule]
  D --> F4[Feature: decide, events, metrics]

Web Experimentation mapping

The current official Web Experimentation workflow maps the design as follows:

Design field	Web Experimentation object
Where treatment can occur	Target By URL or Target By Saved Pages; page triggers and conditions where needed
Eligible population	Audience conditions
Control and treatment	Original and variation under Design > Variations
Random assignment	Design > Traffic Allocation
Primary and supporting outcomes	Track > Metrics
Pre-launch evidence	Preview, testing, Summary, and pre-launch review

Set total traffic allocation to 100% when every eligible visitor should enter the experiment, then distribute that traffic 50/50 between original and treatment. Optimizely distinguishes the eligible share entering the test (allocation) from the share assigned to each variation (distribution) in its traffic documentation. Freeze both before launch.

Feature Experimentation mapping

Feature Experimentation implements the same design in application code and a flag ruleset:

Design field	Feature Experimentation object
Where treatment can occur	Flag evaluation call site
Control and treatment	Flag variations and variables
Eligible population	User-context attributes and audience
Random assignment	A/B Test flag rule, allocation, and distribution mode
Exposure	SDK `decide` call and its decision event
Outcomes	Tracked events configured as metrics

Optimizely's official Feature Experimentation getting-started guide describes the sequence: create a flag, implement it with decide, define an audience and flag rule, then analyze the Experiment Results page. The newer A/B test configuration guide adds the hypothesis, test plan, traffic allocation, metrics, distribution mode, and flag variations.

const user = optimizelyClient.createUserContext(
  durableUserId,
  {
    market: checkout.market,
    account_tier: checkout.accountTier,
  },
)

const decision = user.decide('checkout_recommendations')

if (decision.variationKey === 'treatment') {
  renderCheckout({ recommendations: decision.variables })
} else {
  renderCheckout({ recommendations: null })
}

The durable user ID implements user-level assignment. Attributes implement declared eligibility. The flag key defines the decision point, while the returned variation selects the control or treatment code path. Track the completed-order event with the same user context so the result can be attributed to the decision.

Preserve the analysis contract in Collaboration

If the organization uses Optimizely Collaboration, store the structured design alongside the work. The official hypothesis documentation provides a Brief tab for a structured plan, a Variations tab for images or links, an Experiment tab for linking the implementation, and History for an activity trail.

Attach the design card, metric definitions, MDE inputs, QA evidence, risks, and assumptions. Collaboration can preserve the contract and make changes visible; it does not repair a design that was never specified.

Run a pre-launch design review

Review the design before launch, when weaknesses can still be fixed without contaminating results. Separate validity from feasibility so “we can build it” is not mistaken for “it will answer the question.”

Validity review

Challenge the causal comparison:

Selection: Are eligibility rules evaluated before treatment, and are exclusions symmetrical?
Spillover: Can treated units alter control outcomes?
Attrition: Can missing outcomes differ by variation?
Novelty: Could a temporary reaction dominate the intended long-run effect?
Instrumentation: Can assignment, exposure, and outcomes be reconciled?
Concurrent tests: Can another experiment interact with this treatment or metric?
Identity: Can one randomization unit receive multiple IDs or variations?

Record mitigations and residual risks. Randomization balances expected pre-treatment differences; it does not fix differential loss, broken event collection, or post-assignment filtering.

Feasibility review

Confirm the operating plan:

Required sample and realistic calendar duration.
Engineering, design, analytics, and QA ownership.
Event latency and conversion window.
Rollout and rollback procedure.
Safety monitoring and incident owner.
Browser, device, and environment coverage.
Production baseline freeze and change control.

If the design cannot be executed as written, revise it before exposure rather than improvising after launch.

One-page experiment-design checklist

Decision, causal question, and hypothesis are explicit.
Population, randomization unit, and analysis unit are distinct.
Control and treatment are versioned production specifications.
Eligibility, exclusions, and prior-exposure rules are frozen.
One primary metric has a complete formula and conversion window.
Secondary metrics explain mechanism; guardrails have veto rules.
Assignment, exposure, and outcome events can be reconciled.
MDE, sample, duration, and stopping rule are recorded.
Optimizely pages or flags, audiences, traffic, variations, and metrics match the design.
QA evidence, rollback owner, and launch approval are documented.

Use the Optimizely experiment QA checklist to turn the approved design into a production launch review.

Frequently asked questions

What is experiment design in A/B testing?

Experiment design is the specification that connects a decision to a causal comparison. It defines the population, randomization unit, eligibility, control, treatment, metrics, power assumptions, duration, and decision rule before traffic is exposed.

Is the randomization unit the same as the analysis unit?

Not always. A test can randomize accounts and observe users or transactions, but the analysis must account for dependence within each randomized account. Treating correlated observations as independently randomized overstates the amount of information in the test.

Should an A/B test change only one page element?

It should test one causal idea, which may require several coordinated implementation changes. If the decision requires isolating the contribution of individual elements or their interactions, use separate experiments or a multivariate design.

Can traffic allocation change after an experiment starts?

Operationally, Optimizely permits traffic changes, but they can affect bucketing and interpretation. Predefine allocation and distribution, avoid changing them after results accumulate, and document any safety-driven intervention.

Does Optimizely replace an experiment design document?

No. Optimizely implements pages or flags, audiences, variations, traffic rules, and metrics. The team must still define the causal question, unit, estimand, MDE, duration, risks, and action threshold. The configuration should be a faithful implementation of that prior design.

Use a multivariate test when one decision depends on the separate effect of multiple components.

Running Multiple A/B Tests at Once: When Overlap Is Safe

David Sert — Thu, 20 Aug 2026 05:04:11 +0000

Every experimentation program hits this question within its first few months: test A is running on the homepage, the checkout team wants to launch test B — do we wait? Run them together? Will one contaminate the other? Teams that answer "always isolate" grind their velocity to a halt. Teams that answer "never worry" eventually ship a result that was quietly distorted by a colliding experiment.

The correct answer is more useful than either extreme: overlap is usually fine, the risky cases are identifiable in advance, and when you do need isolation, Optimizely gives you exclusion groups to enforce it deterministically. This guide covers when overlapping A/B tests are safe, when they are not, and exactly how to configure mutually exclusive experiments in both Optimizely Web Experimentation and Feature Experimentation.

The Short Answer: Overlap Is Usually Safe

By default, Optimizely lets experiments overlap — a single visitor can be in test A and test B at the same time. Bucketing uses an independent hash per experiment, so being in A's variation says nothing about which variation of B you will get.

That independence is exactly what makes overlap statistically tolerable. Because visitors from A's control and A's variation flow into B's control and B's variation proportionally, any influence test A has on behavior spreads evenly across test B's arms. It adds a little noise, but it does not systematically favor one of B's variations. For most pairs of experiments — different pages, different flows, different metrics — the interaction risk is negligible and forcing isolation would only slow both tests down.

This is also Optimizely's own guidance: make experiments mutually exclusive only when required, because every exclusion group splits your traffic and extends how long every test inside it needs to run.

What Is an Interaction Effect?

An interaction effect occurs when a user's exposure to one experiment changes how they respond to another, in a way that does not average out. The classic failure mode is two treatments that are individually fine but incoherent together.

A concrete example: test A changes your homepage hero to emphasize a discount. Test B redesigns the checkout to remove the promo-code field. A visitor who saw A's discount messaging and then hits B's variation checkout — where the promised promo field is gone — abandons at an elevated rate. That abandonment shows up in B's results as "the redesign loses," when in reality the redesign only loses in combination with A's variation. Neither team can see this from their own results page.

Interactions distort in proportion to how strongly the two experiences collide. Two tests on unrelated pages measured on unrelated metrics essentially cannot produce this pattern; two tests mutating the same funnel step almost cannot avoid it.

When Overlap Is Risky: Three Red Flags

Optimizely's guidance identifies the situations where interaction effects become likely. Treat these as your pre-launch review checklist for any pair of concurrent tests:

Red flag	Why it matters	Example
Same page or application area	Both treatments compose visually and functionally — variations can conflict outright	Two tests both modifying the pricing page
Same user flow	An upstream change alters who reaches the downstream test and in what state	A signup test and an onboarding test in one funnel
Shared conversion metric	Both tests claim credit for movement in the same number	Two tests both optimizing checkout completion

If none of the three apply, run the tests concurrently and spend your energy elsewhere. If one or more apply, you have a decision to make — and it has more than two options.

Your Four Options for Colliding Tests

flowchart TD
    A{Same page, funnel,<br/>or primary metric?} -- No --> B[Run concurrently<br/>no isolation needed]
    A -- Yes --> C{Same element,<br/>same hypothesis area?}
    C -- Yes --> D[Combine into one test<br/>with more variations]
    C -- No --> E{Can one wait?}
    E -- Yes --> F[Run sequentially]
    E -- No --> G[Exclusion group<br/>mutual exclusion]

Run concurrently. The default. Full traffic to both tests, fastest combined learning.
Combine into one experiment. If two ideas target the same element toward the same goal, they are not two tests — they are one test with more variations. This also measures the combination explicitly instead of leaving it to chance.
Run sequentially. Zero interaction risk and zero configuration, at the cost of calendar time. Sensible when one test is short or clearly higher priority.
Mutual exclusion. Both tests run simultaneously, but no visitor sees both. The traffic cost is real: two tests in a 50/50 exclusion group each get half your traffic, so each takes roughly twice as long to reach significance.

Mutual Exclusion in Optimizely Web: Exclusion Groups

In Web Experimentation, mutual exclusion is implemented through exclusion groups. The mechanics are worth understanding precisely, because they explain the operating rules that follow:

Groups are evaluated before page activation and audience targeting. The moment a visitor touches the snippet, Optimizely deterministically assigns them to one experiment's slice of the group — before checking whether they qualify for that experiment's URL or audience conditions.
Allocation is by weight. If Experiment A has 70% of the group and Experiment B has 30%, every visitor lands in exactly one slice. A visitor in A's slice can never enter B, and vice versa.
Assignment is sticky and deterministic. Returning visitors re-enter the same slice; each group, experiment, and campaign uses its own bucketing ID.
Group membership does not guarantee participation. A visitor allocated to Experiment A's slice must still pass A's page and audience conditions to actually enter the experiment. If they fail, they see nothing — they do not fall through to Experiment B.

That last point is the subtle one: exclusion groups spend traffic on visitors who never enter any experiment. Combined with the slice split itself, this is why Optimizely recommends using groups only when genuinely required.

One shortcut worth knowing: if two experiments already target mutually exclusive audiences — "Android users" and "iPhone users" — they can never overlap, and no exclusion group is needed.

Mutual Exclusion in Feature Experimentation

Feature Experimentation implements the same concept with a cleaner surface. Exclusion groups live in their own tab on the Flags dashboard, and no SDK changes are required — the Decide method automatically respects group membership when assigning users.

To set it up: create the group under Flags → Exclusion Groups → Create New Exclusion Group (name it, pick the environment), then attach each experiment from its flag rule: Configure Rule → "Add this experiment to the following exclusion group", and set the percentage of the group's traffic that experiment receives.

One detail matters for debugging surprises: in FX's bucketing order, exclusion groups are evaluated after forced variations, user allowlists, the user profile service, and audience targeting — but before traffic allocation. A forced variation or allowlist entry will happily put a user into an experiment their exclusion-group slice says they should never see. If a "mutually exclusive" user somehow appears in two experiments, check for forced bucketing first.

The Rules That Keep an Exclusion Group Valid

Exclusion groups only deliver their guarantee if the group's composition stays fixed for its entire lifetime. Optimizely is explicit about the failure modes:

Start every experiment in the group together, and never add one mid-flight. Adding an experiment to a group that is already running shifts the traffic allocation ranges. A visitor who was bucketed into Experiment A can be silently reassigned to Experiment B's slice — they have now been exposed to both, which is precisely what the group existed to prevent.
Never remove a running experiment either. The remaining experiments' bucketing ranges shift to fill the gap, reassigning visitors between experiments and contaminating both result sets.
Duplicating an experiment drops its group membership. A copied experiment starts with no exclusion group attached — easy to miss if you build tests by duplication.
Experiments in the group do not need to **end**** together** — but the group must contain all of them from the first launch until the last one finishes.

Treat an exclusion group like a pre-registered decision: configured before launch, frozen during flight.

How to Detect an Interaction Effect After the Fact

Suppose two tests ran concurrently without isolation and you now suspect a collision. Detection is possible but harder than prevention:

Segment one experiment's results by the other's variations. If Experiment B's lift differs materially between visitors in A's control and A's variation, you have evidence of an interaction. In Optimizely Web you can debug live assignment with ?optimizely_log=debug (look for Group entries) or inspect window.optimizely.get('data').groups, whose weightDistributions property shows the group's allocation.
Expect to be underpowered. Splitting an experiment's sample four ways (A-control/B-control, A-control/B-variant, ...) slashes your sensitivity — a real interaction can easily hide below the noise floor. An inconclusive segmentation is not proof of safety. This is the same power arithmetic covered in choosing a minimum detectable effect.
Check validity first. Before attributing a weird result to an interaction, rule out the boring explanations: a sample ratio mismatch or a peeking-driven false positive is far more common than a true interaction effect.

The honest conclusion from the detection math: if a collision would genuinely matter, design it away up front — with a group, a sequence, or a combined test — rather than hoping to catch it in analysis.

Common Pitfalls

Isolating everything. The most expensive failure mode. A program that puts every test in one giant exclusion group divides its traffic across all of them and multiplies every test's duration. Reserve isolation for the three red flags.
Forgetting the traffic bill. A 50/50 exclusion group doubles both tests' runtimes. Before creating one, check whether your testing velocity can afford it — sequential execution is sometimes faster in wall-clock terms.
Editing a live group. Adding or removing experiments mid-flight reassigns visitors and quietly breaks both experiments. If a test must join, wait for the group's current experiments to finish.
Assuming group membership means participation. Visitors allocated to an experiment's slice who fail its audience conditions see nothing. Budget for that loss when sizing the group.
Trusting exclusion against forced bucketing (FX). Forced variations, allowlists, and the user profile service override exclusion groups in the bucketing order. Audit these before concluding your isolation is broken.

The Decision in One Paragraph

Let concurrent tests overlap by default — independent bucketing spreads any influence proportionally, and the cost of isolation is real traffic and real calendar time. Escalate to mutual exclusion only when two tests share a page, a funnel, or a primary metric, and consider combining or sequencing them before reaching for a group. When you do create an exclusion group — in Web via exclusion groups, in Feature Experimentation via the Flags dashboard — freeze its membership from first launch to last finish. Isolation is a scalpel, not a default.

Sequential Testing in A/B Tests: When Peeking Is Safe

David Sert — Tue, 18 Aug 2026 05:11:19 +0000

You launch a test, and by mid-afternoon the variation is up 8% with a green "statistically significant" badge. The obvious question is also the dangerous one: can you call it now, or do you have to wait? On most classical A/B testing setups the honest answer is that you were never supposed to be looking yet, and acting on that early peek quietly wrecks your error rate. Optimizely's Stats Engine is built specifically so that the answer flips to "yes, you can trust this look" — and the reason why is worth understanding before you ship a decision.

Why peeking breaks a fixed-horizon test

A classical, fixed-horizon A/B test (the kind powered by a t-test or z-test) makes one promise: if you fix the sample size in advance, collect exactly that many visitors, and test once at the end, then your false-positive rate is capped at your significance level — typically 5%. Every part of that promise depends on testing a single time at a pre-committed sample size.

Peeking violates the promise. Each time you evaluate significance on accumulating data, you give random noise another independent chance to cross the threshold. The p-value of an A/A test (two identical variations) wanders up and down as visitors trickle in; over a long enough experiment it will dip below 0.05 purely by chance, even though there is nothing to find. Stop the moment it does, and you have "found" a winner that does not exist. This is the optional stopping problem, and it is a specific instance of the multiple-comparisons problem: more looks means more opportunities to be fooled.

The inflation is not subtle. Optimizely ran the experiment on itself — simulating A/A tests (no real difference) at 5,000 visitors and letting an observer peek. More than 57% of those A/A tests falsely declared a winner or loser at least once. Checking every 500 visitors produced a false-declaration rate around 26%; checking every 1,000 visitors, around 20%. Even applying a naive post-hoc correction still left error rates hovering near 25%. A test you believed was operating at 5% risk was actually running at four to eleven times that, entirely because someone looked early and acted.

This is why a fixed-horizon test needs a pre-computed sample size, and why our sample size calculator exists: under fixed-horizon rules, the sample size is a contract you sign before the experiment starts and are not allowed to renegotiate by watching the results.

See the peeking problem in a short simulation

If the effect feels abstract, simulate it. The script below runs many A/A experiments — control and treatment drawn from the same 10% conversion rate — and checks a standard two-proportion z-test at 20 interim looks, declaring a "winner" the first time p < 0.05.

import numpy as np
from scipy import stats

def peeked_aa_test(n_max=10000, looks=20, alpha=0.05, rng=None):
    """One A/A experiment. Returns True if it EVER crosses significance."""
    rng = rng or np.random.default_rng()
    p = 0.10  # identical conversion rate for both arms — there is no real effect
    control = rng.random(n_max) < p
    treatment = rng.random(n_max) < p
    for n in np.linspace(n_max // looks, n_max, looks).astype(int):
        c_sum, t_sum = control[:n].sum(), treatment[:n].sum()
        pooled = (c_sum + t_sum) / (2 * n)
        se = np.sqrt(pooled * (1 - pooled) * (2 / n))
        if se == 0:
            continue
        z = (t_sum / n - c_sum / n) / se
        p_value = 2 * (1 - stats.norm.cdf(abs(z)))
        if p_value < alpha:      # stop the first time it "looks significant"
            return True
    return False

rng = np.random.default_rng(42)
trials = 5000
false_positives = sum(peeked_aa_test(rng=rng) for _ in range(trials))
print(f"20 peeks: false-positive rate = {false_positives / trials:.1%}")
# A single look at the final sample would sit near 5%.
# Peeking 20 times typically pushes this into the ~20–30% range.

The single-look version of this test behaves: evaluate only at n_max and the false-positive rate sits near 5%, exactly as advertised. Add the interim looks and it climbs into the 20–30% range — the same phenomenon Optimizely measured on real traffic. Nothing about the data changed; only the number of looks did.

Fixed-horizon, Bayesian, and sequential testing

There are three broad ways to analyze an experiment, and Optimizely supports all three. A baking analogy (Optimizely's own) makes the trade-off concrete before the mechanics do:

Frequentist (fixed-horizon) — You must decide the bake time before the cake goes in. You wait for the timer; opening the oven early and pulling the cake out ruins it, even if it looks done, because it may be raw inside.
Bayesian — You can open the oven whenever you like. Each look updates your belief about whether the cake is done, and you act on the probability accumulated so far. You are never told the cake is definitively "finished" — only how likely it is.
Sequential — You can open the oven whenever you like and get a trustworthy verdict each time. If it looks done, it is done.

The distinction that matters for peeking:

Property	Frequentist (fixed-horizon)	Bayesian	Sequential
Sample size calculation	Required before starting	Not required	Not required
Peeking mid-test	Not allowed	Allowed	Allowed
Early stopping	Not allowed	Allowed	Allowed
Result style	Significance / confidence intervals	Probability statements	Frequentist-style significance and confidence intervals

Optimizely's Stats Engine is the sequential option. It gives you the continuous-monitoring freedom of a Bayesian approach while still reporting the frequentist-style significance and confidence intervals most experimentation teams already reason about. For a fuller treatment of the engine itself, see How the Optimizely Stats Engine works.

If you are weighing fixed-horizon, Bayesian, and sequential analysis for a specific test, the stats method picker walks you through the trade-offs.

flowchart TD
  A[Starting an experiment] --> B{Can you fix a sample size in advance and not look until it is reached?}
  B -->|Yes, and you have strong stats expertise| C[Frequentist fixed-horizon]
  B -->|No, you want to monitor continuously| D{Which result style do you prefer?}
  D -->|Direct probability statements| E[Bayesian]
  D -->|Significance and confidence intervals| F[Sequential - Optimizely Stats Engine]

What always-valid inference actually means

The property that makes peeking safe has a precise name: always-valid inference. A p-value (or confidence interval) is "always valid" if it holds its guarantee no matter when you choose to look and stop. With an always-valid p-value, you can check the test after every visitor, stop the first time it drops below your threshold, and your Type I (false-positive) error is still bounded by that threshold. The decision rule is safe under optional stopping by construction.

The mechanism, drawn from the academic work behind Stats Engine (Johari, Pekelis, and Walsh's Always Valid Inference, and the KDD paper Peeking at A/B Tests), is a mixture Sequential Probability Ratio Test (mSPRT), an idea tracing back to Robbins in 1970. Rather than computing a fresh single-shot p-value at each look, the engine maintains a running likelihood ratio that accumulates evidence across the whole stream of visitors. The always-valid p-value is essentially the reciprocal of that accumulated evidence, and the test is what Optimizely calls a "test of power one": the p-value now represents the chance the experiment will ever cross the significance threshold under the null. Because that quantity already accounts for every possible future look, looking early costs you nothing.

Note the attribution: mSPRT is the term used in the underlying literature. Optimizely's product documentation describes the same mechanism in plainer language ("an average likelihood ratio calculated every time a new visitor triggers an event") without naming it.

How sequential testing preserves the error rate

The fixed-horizon test fails under peeking because its 5% guarantee was only ever a promise about a single look. The sequential test makes a stronger promise from the start: the error bound holds across the entire sequence of looks, treated as one continuous procedure. Instead of spending your whole 5% error budget at one pre-chosen moment, the engine spreads and controls it across all looks simultaneously.

The visible consequence, documented by Optimizely, is that statistical significance in a stable experiment should generally increase over time as evidence accumulates, rather than fluctuate. Early on — when the sample is small — large observed gaps are treated conservatively, because a big early swing is exactly what noise produces. As the same-direction difference persists over more visitors, the engine becomes willing to call it. Two forms of evidence drive significance up: larger conversion-rate differences, and differences that persist across more visitors.

Real traffic is not a clean simulation, so significance can occasionally fall. Optimizely reports this happens in roughly 4% of experiments, from two causes: an early run of data that later looks like noise, or a genuine mid-experiment change in conditions. For the latter, Stats Engine has a protective stats reset that can drop significance sharply (potentially to 0%) when it detects that the underlying data-generating process has shifted. That is the safety mechanism doing its job, not a bug. Reducing metric variance up front — for example with CUPED variance reduction — is the cleaner way to reach a confident verdict sooner, rather than hoping an early peek holds.

False discovery rate control across metrics and variations

Peeking over time is one multiplicity problem. Testing many metrics and variations at once is another, and Optimizely handles it with a second layer: false discovery rate** (FDR) control**.

Classical significance controls the false-positive rate across all goals and variations. But you do not implement all of them — you implement the ones that won. Optimizely's documentation gives the sharp example: nine inconclusive results and one false winner is a 10% false-positive rate across everything, but if you only ship winners and one of two winners is false, your rate of shipping a false result is 50%. That "share of your declared winners that are actually null" is the false discovery rate, and it is the number that maps to a real business mistake.

Stats Engine controls FDR using a tiered Benjamini-Hochberg procedure:

Your primary metric is evaluated independently of the others, so it retains full statistical power and reaches significance as fast as possible.
Secondary metrics (ranked 2–5) have their thresholds adjusted for the number of metrics and variations; adding more of them slows each one down but never slows the primary.
Monitoring metrics (ranked beyond 5) each get a fractional weight of 1/n, so they contribute diagnostic signal without dragging on the metrics you actually decide on.

The practical takeaway: rank your metrics deliberately. The metric your decision hinges on belongs in the primary slot. One important caveat from the docs — FDR control is not maintained when you segment results. The deeper you slice, the more the false-positive risk climbs, so treat segments as exploration, not as a basis for decisions.

flowchart LR
  A[New visitor event] --> B[Update running likelihood ratio]
  B --> C[Recompute always-valid p-value and FDR-adjusted significance]
  C --> D{Threshold crossed?}
  D -->|No| E[Keep running - the current read is still valid]
  E --> A
  D -->|Yes| F[Declare winner or loser - safe to stop and act]

When you can stop the test

Here is the direct answer to "am I allowed to stop when it looks significant?" With Optimizely's Stats Engine, yes — a variation that has crossed your significance threshold on the primary metric is a valid result you can act on, with a few practitioner caveats:

Do a launch sanity check first. Statistical validity does not catch a broken implementation. Early in the test, confirm the variation renders correctly, events fire, and traffic split roughly matches your allocation (a sample ratio mismatch usually signals a setup bug, and no statistical engine repairs biased data).
Decide on the metric you ranked primary. Secondary and monitoring metrics are context, not the verdict. See reading the Optimizely results page for how each is displayed.
Weigh practical significance, not just statistical significance. A result can be real and still too small to justify the engineering cost of shipping. Check that the confidence (improvement) interval clears the effect size you actually care about.
Let it run a full business cycle when you can. Statistical validity at any look is not the same as representativeness. If your traffic behaves differently on weekends or across a purchase cycle, a Tuesday-only sample may be valid but unrepresentative. Covering at least one full weekly cycle guards against novelty and day-of-week effects. (This is practitioner best practice, not a statistical requirement of the engine.)

Within those guardrails, you do not owe the test a pre-computed sample size, and you are not penalized for looking. That is the entire point of sequential inference.

Common misconceptions

"Peeking is always cheating." Peeking breaks fixed-horizon tests. Under an always-valid method it is a designed-in feature. The sin is peeking with the wrong statistical engine, not peeking itself.

"Sequential testing is just Bayesian testing." Both allow continuous monitoring, but they answer different questions. Bayesian testing reports the probability a variation is better given a prior; sequential testing reports frequentist significance and confidence intervals that stay valid under optional stopping. Optimizely's Stats Engine is sequential, not Bayesian.

"Significance dropping means the tool is broken." In a stable experiment, significance generally climbs. A drop usually reflects a real change in conditions or an early false signal correcting itself — and a stats reset is the safeguard, not a defect.

"Always-valid means I can ignore sample size entirely." You can skip the pre-commitment to a sample size, but low-traffic tests still take real time to accumulate evidence, and tiny true effects still need many visitors. Always-valid inference removes the peeking penalty; it does not remove the need for data.

"I can slice the winning segment and trust it the same way." FDR control is not maintained across segments. A segment that looks like a standout is a hypothesis for a future test, not a result to ship.

The short version: with a fixed-horizon test you must set the timer before you bake and never open the oven. With Optimizely's sequential Stats Engine you can open the oven whenever you like — and when the result looks done, it genuinely is.

Multi-Armed Bandit Testing: How It Works and When to Use

David Sert — Tue, 18 Aug 2026 05:10:43 +0000

A multi-armed bandit is an algorithm that decides how to split traffic across several variations while a test is still running, shifting more visitors toward whichever option is performing best. Instead of holding an even split until a fixed sample size is reached, a bandit continuously reallocates traffic to earn conversions during the test itself. This guide explains how multi-armed bandits work, the core algorithms behind them (epsilon-greedy, Thompson sampling, and Upper Confidence Bound), the explore-exploit tradeoff that governs every bandit, when a bandit beats a fixed A/B test, and how Optimizely implements bandit optimization.

What Is a Multi-Armed Bandit?

The name comes from slot machines. A single slot machine is nicknamed a "one-armed bandit" because it has one lever and it tends to take your money. Now imagine a row of slot machines, each with a different unknown payout rate. You have a fixed number of pulls and one goal: win as much as possible. Every pull forces a dilemma. Do you keep pulling the machine that has paid out best so far, or do you try the others in case one of them is secretly better? That is the multi-armed bandit problem, and it maps directly onto experimentation. Each "arm" is a variation, each "pull" is a visitor, and the "payout" is a conversion.

A fixed A/B test answers this dilemma by refusing to answer it during the test. It sends an equal share of traffic to every variation until enough data accumulates to declare a winner with statistical confidence, then you deploy the winner afterward. A multi-armed bandit answers it continuously. As evidence builds that one variation converts better, the bandit routes more traffic to it in real time, so fewer visitors are sent to the underperforming options while the test runs.

This changes the objective. A fixed A/B test optimizes for learning: it wants a precise, unbiased estimate of how each variation performs so you can make a confident decision. A bandit optimizes for earning: it wants to maximize total conversions across the whole population of visitors, even at the cost of a cleaner statistical readout. Understanding that tradeoff is the key to knowing when to reach for a bandit.

The Explore-Exploit Tradeoff

Every bandit balances two competing impulses on every decision:

Exploit – Serve the variation that currently looks best, to capture conversions now.
Explore – Serve other variations, to gather more data in case the current leader is not actually the best.

Pure exploitation is dangerous. If an inferior variation happens to get lucky in its first few hundred visitors, a greedy algorithm that only exploits will lock onto it and never discover its mistake. Pure exploration is wasteful: it is just a fixed A/B test that keeps sending traffic to known losers. A good bandit algorithm blends the two, exploring enough to stay confident about which arm is best while exploiting enough to bank conversions along the way.

The metric that formalizes this is regret: the cumulative difference between the conversions you would have earned if you had always served the truly best variation and the conversions you actually earned. A bandit's job is to minimize regret. Early on, regret accrues quickly because the algorithm does not yet know which arm is best. As evidence accumulates and the bandit concentrates traffic on the leader, the rate of regret slows. The whole design of a bandit algorithm is a strategy for driving regret as low as possible, as fast as possible.

The allocation loop below runs continuously for the life of a bandit experiment:

flowchart LR
  A[Split traffic evenly] --> B[Serve a variation]
  B --> C[Observe conversion outcome]
  C --> D[Update value estimate per arm]
  D --> E{Explore or exploit?}
  E -->|Explore| F[Sample under-tested arms]
  E -->|Exploit| G[Send more traffic to the leader]
  F --> B
  G --> B

Because the loop never stops updating, a bandit adapts if conditions change mid-test. If a variation that was losing starts winning (for example, because a new audience arrives), the value estimates shift and traffic follows. A fixed A/B test cannot do this: its allocation is frozen for the duration.

Bandit Algorithms: Epsilon-Greedy, Thompson Sampling, and UCB

"Multi-armed bandit" names the problem, not a single algorithm. Several algorithms solve it, and they differ mainly in how they decide to explore versus exploit. Three are worth knowing.

Epsilon-Greedy

Epsilon-greedy is the simplest bandit. You pick a small exploration rate, epsilon (for example, 0.1). On each visitor, the algorithm flips a biased coin:

With probability epsilon, it explores by serving a random variation.
With probability 1 minus epsilon, it exploits by serving the variation with the highest observed conversion rate so far.

With epsilon set to 0.1, roughly one visitor in ten sees a random variation and the other nine see the current leader. Epsilon-greedy is easy to reason about and easy to implement, but it has two weaknesses. It explores blindly, wasting some of its exploration budget on variations that are clearly bad, and a fixed epsilon keeps exploring at the same rate forever even after the best arm is obvious. A common fix is epsilon-decreasing, where epsilon shrinks over time so the algorithm explores heavily at first and exploits more as confidence grows.

Thompson Sampling

Thompson sampling is a Bayesian approach and the one most modern platforms favor. Instead of tracking a single conversion-rate estimate per variation, it maintains a full probability distribution over each variation's true conversion rate. For a binary conversion metric, that distribution is a Beta distribution updated from the counts of conversions and non-conversions each arm has seen.

On each decision, Thompson sampling draws one random sample from every arm's distribution and serves whichever arm produced the highest sample. The elegance is that exploration falls out automatically. An arm with little data has a wide distribution, so it sometimes produces a high sample and gets explored; an arm with lots of data has a narrow distribution tightly centered on its true rate. Over many decisions, each arm receives traffic roughly in proportion to the probability that it is the best arm. No epsilon to tune, and exploration self-adjusts as certainty grows.

# Illustrative only - demonstrates the Thompson Sampling concept,
# not any specific platform's production implementation.
import random

# (conversions, non_conversions) observed per variation
arms = {
    "A": [12, 388],   # 12 conversions in 400 visitors
    "B": [30, 370],   # 30 conversions in 400 visitors
    "C": [18, 382],
}

def choose_variation(arms):
    best_arm, best_sample = None, -1.0
    for name, (conv, non_conv) in arms.items():
        # Sample the arm's plausible conversion rate from its Beta posterior
        sample = random.betavariate(conv + 1, non_conv + 1)
        if sample > best_sample:
            best_arm, best_sample = name, sample
    return best_arm

# Arm B wins most draws, but A and C are still explored
# whenever their sampled rate happens to come out on top.

Upper Confidence Bound (UCB)

UCB follows the principle of "optimism in the face of uncertainty." For each arm it computes the observed conversion rate plus a bonus that grows with how uncertain that estimate is. An arm that has been shown to few visitors carries a large uncertainty bonus, so UCB is optimistic about it and explores it; an arm shown to many visitors carries a small bonus, so it is judged mostly on its actual performance. On each decision, UCB simply serves the arm with the highest upper confidence bound. As every arm accumulates data, the bonuses shrink and the algorithm converges on exploiting the genuinely best arm. Unlike Thompson sampling, UCB is deterministic given the same data, which makes its behavior easy to audit.

Bandit vs Fixed A/B Test

A bandit and a fixed-horizon A/B test are built for different jobs. The table below summarizes the practical differences.

Dimension	Fixed A/B test	Multi-armed bandit
Primary objective	Learn which variation is best (unbiased estimate)	Maximize conversions during the test
Traffic allocation	Fixed and even until conclusion	Dynamic; shifts toward the leader in real time
Statistical significance	Reaches significance and quantifies the effect	Does not target or report significance
Effect-size estimate	Precise, with confidence intervals	Biased toward the winner; weaker per-arm reads
Best for	Durable decisions and roadmap learnings	Short-lived opportunities; many low-stakes options
Losing variations	Keep receiving traffic until the test ends	Starved of traffic as evidence accumulates

The core trade is precision for opportunity cost. A fixed A/B test pays a known price - it deliberately sends half its traffic to a variation that may be worse - in exchange for a clean, trustworthy measurement of exactly how much better the winner is. A bandit reduces that opportunity cost by steering traffic to the leader early, but it sacrifices the clean measurement: because allocation is entangled with performance, you cannot read an unbiased effect size off a bandit the way you can off a controlled A/B test. If you plan test duration with a sample-size and statistical power calculation and care about the minimum detectable effect, you are thinking in fixed-A/B-test terms, and that is usually the right default.

When a Multi-Armed Bandit Beats a Fixed A/B Test

A bandit is the better tool when maximizing conversions during the test matters more than measuring the exact difference between variations. Reach for a bandit when:

The opportunity is short-lived. For a weekend sale, a holiday banner, or a news-driven promotion, there is no "afterward" in which to deploy the winner. Whatever conversions you earn, you earn during the campaign. A bandit captures them; a fixed A/B test spends the whole window collecting evidence you will never act on.
You are choosing among many low-stakes options. Testing ten subject lines or fifteen hero images as a full A/B test needs enormous traffic to power every comparison. A bandit concentrates traffic on the front-runners and quietly starves the losers, earning more conversions without waiting for every arm to reach significance.
You do not need a defensible effect size. If the decision is "serve the best headline," not "prove headline B lifts conversions by 4.2% for the quarterly readout," the bandit's biased estimates are an acceptable price.

Stick with a fixed A/B test when the opposite is true: when you need a trustworthy, quantified result for a durable decision, when the change is high-stakes or hard to reverse, when you want a clean learning to generalize to future work, or when peeking and early stopping are the real concern - in which case sequential testing is the tool designed for valid early looks, not a bandit. The decision comes down to a single question.

flowchart TD
  A[New test] --> B{Need a precise, defensible<br/>effect size?}
  B -->|Yes| C[Fixed A/B test]
  B -->|No| D{Is the window short<br/>or are there many options?}
  D -->|Yes| E[Multi-armed bandit]
  D -->|No| C

How Optimizely Implements Multi-Armed Bandits

Optimizely groups its experiment types by objective. A/B tests, multivariate tests, and Stats Accelerator are built for learning and incorporate statistical significance, while multi-armed bandit and contextual bandit optimizations are built for immediate impact and optimize traffic allocation without performing significance analysis. Optimizely states plainly that a multi-armed bandit "does not rely on a fixed sample size or equal traffic allocation," and that bandits "do not achieve statistical significance because their goal is to optimize traffic allocation dynamically to maximize conversions." (Optimizely: Contextual bandits FAQ)

Multi-armed bandit optimization

In a standard Optimizely multi-armed bandit, you choose a primary metric and the platform displays variations to visitors, then routes more traffic toward whichever variation has the greatest impact on that metric. Optimizely's own example is capitalizing on conversions "during a short cycle like a weekend sale" - exactly the short-window case where a bandit shines. (Optimizely: Contextual bandits) Because a MAB optimizes for impact rather than significance, the Results page shows an Improvement figure for the variation over the baseline, but it does not declare a statistically significant winner the way a manual A/B test does. (Optimizely: Experiment Results page) Bandits also do not use the sticky bucketing that keeps an A/B-test visitor in the same variation across sessions, since a bandit is continuously reallocating traffic.

Exploration and exploitation rates

Optimizely exposes the explore-exploit tradeoff directly as two rates. The exploration rate is the probability the system serves a random variation (Optimizely calls this the "learning rate"), and the exploitation rate is the probability it serves the best-performing variation. When set to automatic, the exploration rate starts high and decreases as the model gathers more events - the same epsilon-decreasing idea described above, managed for you. (Optimizely: Contextual bandits FAQ)

Stats Accelerator and contextual bandits

Optimizely offers two neighboring tools that are easy to confuse with a plain bandit. Stats Accelerator reallocates traffic to help an experiment reach a statistically significant result faster - it optimizes for learning speed, not pure earning, and it keeps significance in play. A contextual bandit (CMAB) goes the other direction: rather than finding one best variation for everyone, it personalizes the variation per visitor using user attributes, running server-side behind a prediction API. Like a plain MAB, a contextual bandit does not report statistical significance. (Optimizely: Contextual bandits)

Choosing an Allocation Method

The right choice depends on whether you are optimizing for learning, earning, or personalization. Because Optimizely gives you a manual A/B test, Stats Accelerator, a multi-armed bandit, and a contextual bandit, picking among them is its own decision. For a full side-by-side of those four allocation methods and a decision framework for choosing between them, see the companion guide: Stats Accelerator vs MAB vs contextual bandit. If your reason for reaching past a fixed A/B test is speed to a trustworthy result rather than raw conversions, two complementary techniques are worth combining with your A/B tests instead of a bandit: CUPED reduces variance so tests reach significance faster, and Optimizely's Stats Engine provides always-valid significance so you can act on results sooner without inflating false positives.

Frequently Asked Questions

Does a multi-armed bandit reach statistical significance?

No. A multi-armed bandit is designed to maximize conversions by dynamically reallocating traffic, not to test a hypothesis, so it does not rely on a fixed sample size or equal allocation and does not report statistical significance. If you need a defensible, quantified effect size, run a fixed A/B test instead.

Is a multi-armed bandit the same as an A/B test?

No. Both compare variations, but an A/B test holds an even split to measure each variation precisely, while a bandit continuously shifts traffic toward the leader to earn more conversions during the test. An A/B test optimizes for learning; a bandit optimizes for earning.

What is the difference between epsilon-greedy and Thompson sampling?

Epsilon-greedy explores at a fixed rate by serving a random variation a set fraction of the time, exploiting the current best otherwise. Thompson sampling explores adaptively by sampling from each variation's probability distribution, so uncertain variations get explored more and exploration shrinks automatically as data accumulates. Thompson sampling generally produces lower regret without a rate to tune.

When should I not use a multi-armed bandit?

Avoid a bandit when you need a precise effect size for a durable or high-stakes decision, when you want a clean learning to generalize to future tests, or when your primary concern is valid early stopping - sequential testing is the correct tool for peeking, not a bandit.

Does Optimizely support multi-armed bandits?

Yes. Optimizely offers multi-armed bandit optimization that reallocates traffic to the highest-impact variation for a chosen primary metric, plus contextual bandits that personalize the variation per visitor using user attributes. Both optimize for impact and do not report statistical significance.

Key Takeaways

A multi-armed bandit reframes experimentation from "which variation is best?" to "how do I earn the most conversions while I find out?" It answers the explore-exploit tradeoff continuously, shifting traffic toward the leader to minimize regret rather than holding an even split to measure each option cleanly. Epsilon-greedy, Thompson sampling, and UCB are three strategies for making that explore-exploit decision, with Thompson sampling the modern default because its exploration self-adjusts as certainty grows. Reach for a bandit when the opportunity is short-lived or you are screening many low-stakes options and do not need a defensible effect size; stick with a fixed A/B test when you need a trustworthy, quantified result. Optimizely implements bandit optimization directly, exposing exploration and exploitation rates and offering both standard and contextual bandits, neither of which reports statistical significance - because for a bandit, conversions earned, not significance reached, is the goal.

How to Choose a Minimum Detectable Effect (MDE)

David Sert — Tue, 18 Aug 2026 05:08:33 +0000

You have opened your sample-size calculator, entered your baseline conversion rate, kept the defaults for significance and power, and now the calculator is asking for one more number: the minimum detectable effect. It is the input that decides everything downstream — how many visitors you need, how many weeks the test runs, and whether the experiment is even worth starting. Yet it is the one field with no obvious "correct" value, so it tends to get a number typed into it that is really a wish rather than a decision.

This guide is about choosing that number deliberately. Not "what is MDE" in the abstract, but "what should I actually put in the box," why the answer is a business judgement as much as a statistical one, and how to avoid the two mistakes that quietly ruin experiment programs: an MDE so small the test never finishes, and one so large the test is underpowered to see the wins you care about.

What a minimum detectable effect actually is

The minimum detectable effect is the smallest true improvement you want your experiment to be able to reliably catch. Optimizely defines it as "the smallest improvement you are willing to detect" and notes that it "determines how 'sensitive' an experiment is." If you set a 5% MDE, you are telling the calculator: size this test so that a genuine 5% lift would show up as statistically significant. A real effect exactly at your MDE has a good chance of being detected; a real effect smaller than your MDE probably will not, because you did not buy enough traffic to see it.

The critical thing to internalize is that MDE is a planning input, not a result. It is not a prediction of how much your variation will lift conversions, and it is not the effect you will report at the end. It is a design lever you set beforehand to determine sample size. The test itself might measure a 3% lift or a 9% lift or nothing at all — the MDE only governed how sensitive the instrument was.

Three inputs work together to fix your sample size: the baseline conversion rate, the statistical significance and power you require, and the MDE. Hold the first two constant and MDE becomes the dial that trades detectable precision against how much traffic — and therefore time — the test consumes.

Relative vs. absolute MDE

Almost every calculator, Optimizely's included, expresses MDE as a relative lift, and this trips people up constantly. Optimizely states that MDE "represents the relative minimum improvement over the baseline." A 10% MDE does not mean "10 percentage points." It means a 10% change relative to your baseline.

Work through Optimizely's own example. If your baseline conversion rate is 20% and you set a 10% MDE, the test is sized to detect a move outside the absolute band of 18% to 22% — because 10% of 20% is 2 percentage points. So a "10% relative MDE" is a "2 percentage-point absolute MDE" at that baseline.

This distinction matters because absolute and relative framings diverge wildly at different baselines. A 10% relative MDE on a 2% baseline checkout rate is a 0.2 percentage-point move — invisible without enormous traffic. The same 10% relative MDE on a 40% baseline is a 4 percentage-point move, far easier to detect. Always confirm which convention your calculator uses before you type a number, and think in the units your business actually cares about. If stakeholders reason in percentage points ("we need to add two points of conversion"), convert to relative before entering it.

Why sample size scales with the inverse square of MDE

Here is the single most important fact for setting a realistic MDE: required sample size grows with roughly one over the square of the MDE. Optimizely puts the direction plainly — "the smaller your MDE is, the larger the sample size required to reach statistical significance" — and the standard fixed-horizon formula makes the rate precise.

Sample size per variation is approximately proportional to 1/MDE². The consequences are steep and non-linear:

Halving your MDE (say 10% down to 5%) roughly quadruples the sample size, because (1/0.5)² = 4.
Cutting MDE to a third (10% to 3.3%) needs roughly nine times the traffic.
Going the other way, doubling the sample size only shrinks your detectable effect by about 29%, not half, because 1 − 1/√2 ≈ 0.293.

That last point is the one that surprises people. Traffic buys detectability at a punishing exchange rate. You cannot casually "just run it a bit longer" to chase a smaller effect — the marginal week of traffic detects a smaller and smaller increment of lift. This asymmetry is exactly why MDE has to be chosen against what your traffic can afford, not against what you would like to prove.

The 1/MDE^2 penalty (relative MDE, baseline and settings held constant)

Anchor point (from Optimizely's own worked example):
  baseline = 15%,  MDE = 10% relative  ->  ~8,000 visitors PER VARIATION

Apply the sample-size ∝ 1 / MDE^2 scaling law to that anchor:

  MDE = 10%   ->  N          ~   8,000 / variation   (baseline)
  MDE =  5%   ->  N x (10/5)^2  = 4N  ~  32,000 / variation
  MDE =  2.5% ->  N x (10/2.5)^2 = 16N ~ 128,000 / variation

Reading it the other way:
  To HALVE the effect you can detect, you need ~4x the visitors.
  Doubling visitors only shrinks detectable MDE by ~29% (1 - 1/sqrt(2)).

Note: 8,000 is Optimizely's figure; the 32,000 and 128,000 are that
figure scaled by 1/MDE^2 — illustrative, not calculator-exact. Real
Stats Engine (sequential) durations differ, but the scaling holds.

At two visitors per test, the difference between a 10% and a 2.5% MDE is the difference between a test that concludes in a couple of weeks and one that needs several months of the same traffic. Same hypothesis, same page — a 4x to 16x difference in cost, driven entirely by the number you chose for one field.

How to choose an MDE that isn't wishful thinking

A defensible MDE sits at the intersection of two independent constraints. Set it too optimistically on either and the experiment fails you.

Constraint one — the smallest lift worth shipping. Ask what improvement would actually change a decision. If shipping the variation carries engineering cost, maintenance, or risk, a 0.5% lift might not be worth it even if it is real. The floor of your MDE should be the smallest effect that would still make you press "launch." Optimizely frames this as using "potential business impact to decide on the sensitivity of your experiment" — and notes that when conversions tie directly to revenue, a lower MDE can be justified precisely because each fraction of a percent is worth real money. A checkout step feeding millions in revenue earns a smaller MDE than a blog newsletter signup.

Constraint two — what your traffic can power in a reasonable window. Take your weekly traffic to the surface under test, decide the longest you are willing to run (commonly two to four weeks so you capture full business cycles without letting the test rot), and work out the largest sample you can realistically collect. Then find the smallest MDE that sample can power. Optimizely's prioritization guidance is explicit: divide total required sample by the traffic you can allocate to see how long a test will take, and judge the tradeoff — "a 4% lift in 5 weeks may be a reasonable tradeoff of impact for effort, but a 2% lift measured over several months may not be."

Your MDE is the larger of these two floors. If the smallest lift worth shipping is 3% but your traffic can only power an 8% MDE in a month, you have a mismatch to resolve — you cannot honestly detect the 3% you care about. Do not pretend otherwise by typing 3% into a test that cannot see it.

As rough orientation — not a rule — high-traffic pages often support a 2–5% relative MDE, while lower-traffic flows frequently bottom out around 10–20%. Treat these as sanity checks on your own calculation, not targets. And follow Optimizely's advice to work in "limits and ranges rather than looking for exact numbers": run the calculator at a few candidate MDEs and pick the point where detectable precision and runtime both look acceptable. The sample size calculator and velocity calculator turn each candidate MDE into a concrete sample size and a testing-cadence estimate, which is the fastest way to see the tradeoff in weeks rather than abstractions.

A decision framework for picking your MDE

flowchart TD
    A[Start: which MDE do I enter?] --> B[What is the smallest lift<br/>worth shipping? = business floor]
    A --> C[How much traffic can I collect<br/>in an acceptable window?]
    C --> D[Smallest MDE that traffic<br/>can power = traffic floor]
    B --> E{Is business floor<br/>>= traffic floor?}
    D --> E
    E -->|Yes| F[Set MDE = business floor.<br/>Test is worth running and powerable]
    E -->|No: want to detect smaller<br/>than traffic allows| G[Mismatch — pick one]
    G --> H[Reduce variance:<br/>CUPED, better metric,<br/>higher-traffic surface]
    G --> I[Extend runtime or<br/>concentrate traffic allocation]
    G --> J[De-prioritize: effect you care<br/>about is too small to power]
    H --> E
    I --> E
    F --> K[Enter MDE in calculator,<br/>commit sample size, launch]

When you hit the mismatch branch, you have real options before you give up. Variance reduction is the highest-leverage one: techniques like CUPED lower the noise in your metric, which effectively lets you detect a smaller effect from the same traffic — the equivalent of shrinking your MDE for free. You can also choose a higher-traffic surface, pick a metric closer to the change, or concentrate allocation. Only when none of those close the gap should you de-prioritize, which is itself a legitimate and common outcome — Optimizely explicitly advises that "if the traffic cost is too high, consider de-prioritizing the hypothesis… or seek to measure lift with less granularity."

To see how much a given pre-period correlation shrinks the sample you need, try the CUPED savings estimator.

Common mistakes: too low and too high

Two symmetric errors account for most botched MDE decisions.

MDE set too low (the test that never ends). Chasing a tiny effect because "even 1% matters" without checking whether your traffic can power it. The 1/MDE² penalty means a 1% MDE can demand a hundred times the sample of a 10% MDE. The test drags on for months, business context shifts underneath it, and pressure builds to peek and stop early — which inflates false positives if you are not on a method built for it. If you genuinely need to detect small effects, plan for it with variance reduction or a sequential-testing approach, don't just lower the number and hope.

MDE set too high (the underpowered test). Entering a large MDE to make the sample size look affordable, then treating a non-significant result as proof the variation "didn't work." It proves no such thing. A test sized for a 15% MDE is simply blind to a real 6% lift — a genuinely valuable win — because it was never powered to see it. You will ship losers and kill winners on the strength of a test that was underpowered by construction. High MDEs are only honest when the smallest lift worth shipping really is that large.

Peeking as an MDE problem in disguise. When a test sized for a modest MDE runs longer than hoped, the temptation is to check daily and stop the moment significance flickers. On a fixed-horizon test this quietly wrecks your error rate. The disciplined fixes are to size honestly up front and, if you need the flexibility to stop early, adopt a method designed for it — see sequential testing and the peeking problem.

MDE is a planning input, not a result

The recurring theme is worth stating once more, cleanly: the MDE you enter shapes the experiment; it is not something the experiment gives back. Optimizely's own guidance is to "use MDE as a guide rather than an exact prediction." It sets the sensitivity of your instrument before you collect a single visitor. Confuse it with the expected or measured lift and you will either over-promise ("we'll get a 10% lift" — no, you sized to detect one) or misread a null result.

So treat the MDE field as a design decision with two owners: the business, who names the smallest lift worth shipping, and the traffic, which caps how small an effect you can afford to detect in a sensible window. Set your MDE at the larger of those two floors, verify the resulting sample size and runtime in a calculator, and only then launch. Done that way, the number in the box stops being a wish and becomes the honest specification of what your experiment can — and cannot — tell you.

Once your MDE is set, size the test with the sample size calculator, pace your roadmap with the velocity calculator, and if the effect you care about is too small to power, reach for CUPED variance reduction or sequential testing before you compromise on the MDE itself.

Reach Significance Faster: CUPED Variance Reduction

David Sert — Tue, 18 Aug 2026 05:08:24 +0000

Experiments stall for one boring reason more than any other: not enough signal relative to noise. When a metric like revenue-per-visitor swings wildly from user to user, the treatment effect you care about is buried under variance that has nothing to do with your change. CUPED (Controlled-experiment Using Pre-Experiment Data) attacks that variance directly — it uses each user's own pre-experiment behavior to subtract out the predictable part of the noise, so the same experiment reaches significance with less traffic and less time. This article explains the intuition, the math, how Optimizely's regression-based implementation differs from the textbook version, and — just as important — the cases where CUPED does nothing at all.

The Problem: Variance, Not Effect Size, Is Usually the Bottleneck

The number of visitors an experiment needs to detect an effect scales with the variance of the metric divided by the square of the effect you want to detect. You cannot manufacture a bigger effect — that is set by the quality of your idea. But you can often shrink the variance, and every reduction in variance translates directly into a smaller required sample size.

This matters most for exactly the teams that struggle: low-traffic sites, high-value pages that convert rarely, and revenue or engagement metrics with long, heavy tails where a handful of whales dominate the average. For those metrics, the classic advice — "run it longer" — is expensive and sometimes impossible. Variance reduction is the alternative: change the statistics, not the traffic.

CUPED is the most widely adopted variance-reduction technique in the industry, introduced by Deng, Xu, Kohavi, and Walker at Microsoft in 2013 and now standard at companies including Netflix, Booking.com, and Optimizely. If you want the broader context on how experiments reach significance in the first place, see OptiPilot's explainer on how the Optimizely Stats Engine works.

The Intuition: Noise You Can Predict, You Can Remove

Here is the core idea in one sentence: if you can predict part of a user's post-experiment metric before the experiment even starts, that predictable part is not evidence about your treatment — so remove it.

Consider revenue-per-user on an e-commerce site. A customer who spent 500 USD in the two weeks before your test is likely to spend more than average during the test, regardless of which variation they land in. That tendency is a property of the user, not your change. When a big spender happens to land in the treatment group, they inflate the treatment average — pure luck of the draw, and exactly the kind of noise that makes results wobble.

Because assignment to control or treatment is random, a user's pre-experiment behavior is (in expectation) balanced across variations and uncorrelated with the treatment assignment. That is what makes it a safe covariate: it is strongly correlated with the outcome, but independent of the thing you are trying to measure. CUPED subtracts the predictable, user-level component of the metric and analyzes what remains. The treatment effect survives untouched; the random noise shrinks.

The strength of the effect hinges entirely on one quantity: the correlation between a user's pre-experiment metric and their in-experiment metric. High correlation (a user's past revenue strongly predicts their future revenue) means large variance reduction. Zero correlation means CUPED does nothing — it cannot hurt you, but it cannot help either.

flowchart LR
    A[Pre-experiment window<br/>e.g. 2 weeks of history] --> B[Per-user covariate X<br/>past value of the metric]
    C[In-experiment metric Y] --> D[Regress Y on X<br/>remove predictable part]
    B --> D
    D --> E[Residuals:<br/>variance reduced]
    E --> F[Stats Engine]
    F --> G[Significance reached<br/>with less traffic / time]

The Math

The classic CUPED estimator (2013)

The original method adjusts the metric mean using a single covariate X (typically the same metric measured in the pre-period). The CUPED-adjusted estimate of the mean is:

Ŷ_cuped = Ȳ − θ (X̄ − E[X])

where θ is chosen to minimize the variance of the adjusted estimator:

θ = Cov(Y, X) / Var(X)

E[X] is the expected value of the covariate across the whole population (the same for both groups, since assignment is random). Subtracting θ(X̄ − E[X]) removes the component of each group's mean that is explained by the covariate. The variance of the adjusted metric is:

Var(Ŷ_cuped) = Var(Ȳ) · (1 − ρ²)

where ρ is the correlation between Y and X. This single equation is the whole story: a covariate correlated 0.7 with the outcome removes 0.7² ≈ 49% of the variance; a covariate correlated 0.3 removes only about 9%. Your variance reduction is the square of the correlation.

Illustrative Python: computing the classic CUPED adjustment

The snippet below computes the classic mean-adjustment CUPED estimate from pre-period and in-experiment arrays. It is intended to build intuition — it is not how Optimizely computes CUPED internally (see the next section), and it uses the pooled covariate to estimate θ, which is the standard practice.

import numpy as np

def cuped_adjust(pre, post):
    """
    Classic CUPED (Deng et al. 2013) mean adjustment.

    pre  : per-user metric value in the pre-experiment window (covariate X)
    post : per-user metric value during the experiment (outcome Y)
    Returns the CUPED-adjusted post values, aligned per user.
    """
    pre = np.asarray(pre, dtype=float)
    post = np.asarray(post, dtype=float)

    # theta = Cov(Y, X) / Var(X), estimated on the pooled sample
    theta = np.cov(post, pre, bias=True)[0, 1] / np.var(pre)

    # Center the covariate on its population mean, then subtract its
    # predictable contribution from each user's outcome.
    adjusted = post - theta * (pre - pre.mean())
    return adjusted, theta

# --- demo: correlated pre/post, random assignment ---
rng = np.random.default_rng(0)
n = 20_000
user_baseline = rng.gamma(shape=2.0, scale=10.0, size=n)   # heavy-tailed "spend"
post = user_baseline + rng.normal(0, 5, size=n)            # in-experiment metric
treatment = rng.integers(0, 2, size=n)
post = post + treatment * 1.0                              # true +1.0 effect

adjusted, theta = cuped_adjust(user_baseline, post)

raw_effect = post[treatment == 1].mean() - post[treatment == 0].mean()
adj_effect = adjusted[treatment == 1].mean() - adjusted[treatment == 0].mean()

print(f"theta = {theta:.3f}")
print(f"raw   variance of post:     {post.var():.2f}")
print(f"cuped variance of adjusted: {adjusted.var():.2f}")
print(f"raw effect:  {raw_effect:+.3f}")
print(f"cuped effect:{adj_effect:+.3f}")   # unbiased: still ~+1.0, lower variance

The adjusted metric has substantially lower variance while the estimated treatment effect stays centered on the truth. That is the entire value proposition: same answer, tighter confidence interval, fewer users required.

Optimizely's Regression-Based CUPED

Optimizely does not use the 2013 mean-adjustment formula directly. Its implementation is a regression-based covariance adjustment, which the documentation describes as "more advanced than the standard CUPED method." Instead of adjusting the mean with a single θ, Optimizely fits a linear regression:

Yᵢ = α + β·Tᵢ + γᵀ·X̃ᵢ + eᵢ

Yᵢ — the numeric target metric for user i
Tᵢ — treatment indicator (0 = control, 1 = treatment)
X̃ᵢ — centered covariates derived from that user's pre-experiment data
α, β, γ — regression coefficients
eᵢ — error term

The coefficient β is the treatment effect, and its Ordinary Least Squares estimate β̂ is an unbiased estimator of the true effect. Including the pre-experiment covariates X̃ᵢ reduces the variance of β̂ — that is the variance reduction, expressed in regression form.

In practice Optimizely runs this in two steps, following the Frisch–Waugh–Lovell theorem:

Regress Yᵢ on the covariates X̃ᵢ and compute each user's predicted value.
Compute the residuals Yᵢ − Ŷᵢ and feed those residuals into the Stats Engine.

The Stats Engine then evaluates significance on the residualized, lower-variance metric.

Why regression rather than the classic formula?

The regression framing is strictly more general. As Optimizely's own documentation notes, if the covariate vector X̃ᵢ contained nothing but the single historical value of Yᵢ, the regression method would collapse exactly to the classic CUPED estimator. Because it can incorporate multiple covariates and handles them in a unified least-squares framework, it dominates the standard method — hence Optimizely's choice. In the current release, the covariates are limited to the pre-experiment calculations of your primary and secondary target metrics; user-defined covariates are not yet supported.

When CUPED Applies — and When It Does Not

CUPED is not a universal switch. Verified against Optimizely's documentation, here is where it helps and where it is silent or unavailable.

CUPED helps when

The metric is numeric (revenue per visitor, pages per session, time on site, order value).
Users have pre-experiment history for that metric.
That history is correlated with in-experiment behavior — the stronger the correlation, the larger the reduction.

CUPED does nothing (or is unavailable) when

The metric is a conversion / binary metric. Optimizely restricts CUPED to numeric metrics; it is most effective there and is not offered for conversion-rate metrics.
Users have no prior data. The pre-experiment window runs from (by default) two weeks before the experiment start date up to each user's first decision event. A user with no activity in that window — a brand-new user, or a newly created metric with no history — contributes no covariate, so CUPED has no effect for them.
The pre-period metric is uncorrelated with the outcome. Zero correlation means zero variance reduction. It will not bias your result, but it will not speed it up.

flowchart TD
    A[Want faster significance?] --> B{Is the metric numeric?}
    B -- No, it is a conversion metric --> X[CUPED not available<br/>consider outlier management instead]
    B -- Yes --> C{Do users have<br/>pre-experiment history?}
    C -- No, new users / new metric --> Y[CUPED has no effect]
    C -- Yes --> D{Is history correlated<br/>with in-experiment metric?}
    D -- Weakly / not at all --> Z[Little to no benefit<br/>but safe to leave on]
    D -- Strongly --> W[Enable CUPED<br/>meaningful variance reduction]

A useful mental model: CUPED rewards metrics where the past predicts the present. Returning-user revenue and engagement metrics are ideal. First-touch acquisition metrics, or anything measured on users with no history, are exactly the cases where it quietly does nothing.

Setting Up CUPED in Optimizely

CUPED lives on Optimizely's new A/B Results page and its warehouse-native Analytics Scorecards. A few prerequisites and steps, verified against the docs:

Prerequisites

Access via Opti ID, on the new A/B Results page.
Manual A/B experiments only. Stats Accelerator experiments and multivariate tests are not supported.
A supported data warehouse: Snowflake, BigQuery, or Databricks (CUPED is part of Warehouse-Native Experimentation Analytics). Contact Optimizely for other warehouses.
Numeric metrics — CUPED options appear only for numeric metrics, alongside outlier management.

Steps

Create an Experiment Scorecard in Analytics (or open the new A/B Results page for your experiment).
Under Advanced options, enable the CUPED toggle.
Optionally set the CUPED duration — the length of the pre-experiment window. The default is two weeks of history; adjust it if a different window better reflects your users' behavior.
Click Run (or Ctrl/Cmd+Enter) to recalculate results with CUPED applied. You can toggle it on and off to compare scorecards side by side.

Because toggling CUPED re-runs the analysis over your warehouse data, treat it as a lens on the same experiment rather than a change to the experiment itself — the underlying decision and conversion events are untouched.

Common Pitfalls

Turning it on and expecting magic on conversion metrics. CUPED is numeric-only. If your primary metric is a conversion rate, CUPED will not appear or apply. For heavy-tailed numeric metrics, pair CUPED with Optimizely's outlier management (Winsorization or capping) — they address different problems (systematic vs. extreme-value variance) and compose well.

Expecting help where there is no history. For a metric that only starts collecting data at experiment launch, or an audience of brand-new users, the covariate is empty and CUPED has no effect. This is not a bug; it is the method's precondition. Plan a longer pre-period, or accept that variance reduction is not available for that metric.

Peeking at "with vs. without CUPED" and cherry-picking. CUPED is a legitimate variance-reduction method configured before you look at results, not a knob to flip after the fact until significance appears. Decide up front that you will report the CUPED-adjusted scorecard, and stick with it. Toggling repeatedly and keeping whichever view crosses the threshold reintroduces exactly the false-positive risk that a disciplined stats process exists to prevent. Continuous monitoring is valid under Optimizely's Sequential Stats Engine in a way it is not under a fixed-horizon test — but that is a property of the statistical method, not a license to cherry-pick which CUPED view you report.

Assuming a covariate-imbalance safety net exists. Optimizely notes there are no automated health checks yet for covariate imbalance, the way there are for Sample Ratio Mismatch. With sparse prior data, an unbalanced covariate can in principle increase variance. In practice this is uncommon, but do not assume the platform will flag it for you.

How Much Time or Traffic Does It Actually Save?

The honest answer: it depends entirely on the correlation between your pre-period and in-experiment metrics, and Optimizely does not publish a guaranteed number. The mechanism, however, is precise. Because variance reduction equals the square of that correlation, and required sample size scales linearly with variance, the fractional reduction in required traffic is approximately the fractional reduction in variance.

The original 2013 CUPED research and subsequent industry practice commonly report variance reductions in the 30–50% range on well-correlated engagement and revenue metrics — which, taken at face value, corresponds to needing roughly 30–50% fewer users (or days) to reach the same significance. Treat that as an illustrative band from the literature, not an Optimizely commitment: on a metric with weak pre-period correlation you might see a few percent, and on a conversion metric you will see nothing because CUPED does not apply.

The practical takeaway for planning: if your metric is numeric and your users have predictive history, CUPED can meaningfully shorten test duration for free — no extra traffic, no change to your experiment design. To translate an expected variance reduction into concrete sample-size and runtime numbers for your own baseline and MDE, use OptiPilot's traffic and sample-size estimator and compare the required duration with and without the reduction.

To translate a specific pre-period correlation into an estimated sample-size and duration saving, use the CUPED savings estimator.

Summary

CUPED reduces the variance of numeric experiment metrics by using each user's pre-experiment behavior as a covariate, stripping out predictable noise so tests reach significance faster without more traffic. Optimizely implements it as a regression-based covariance adjustment — a generalization of the classic 2013 estimator — that residualizes the metric and feeds the result to its Stats Engine. It shines on numeric, high-history, well-correlated metrics; it is unavailable for conversion metrics and inert for users with no prior data. Turn it on for the right metrics, decide to use it before you look at results, and it is one of the cheapest ways to make an under-powered experiment program faster.