A/B testing

Purpose

A/B testing settles a delivery decision on evidence: which variant to ship, by how much it moves the metric that matters and with what certainty. It converts a design or product debate into a measured causal estimate, where the discussion would otherwise stay an arbitration between opinions, often that of the most senior person in the room.

The deliverable is two linked artefacts. A pre-registered design, declared before any collection: the hypothesis, the primary metric, the guardrails, the sample size and the duration. A result table that, for each arm, gives the count, the value of the metric, the gap against control, its confidence interval and the verdict, ship the variant or not. The value of the technique comes from the fact that this gap is caused by the change.

Usage

When to use it

• Isolable and measurable change: the variant switches on by itself and exposes a quantifiable usage indicator, without depending on other modifications.
• Traffic sufficient for power: enough users to detect the target effect within a reasonable horizon.
• Reversible, low-stakes decision: the losing variant stays tolerable for the sample that saw it.
• Arbitration between competing variants: settle a design choice on a measurement rather than on an opinion.

When not to use it

• Traffic too low to reach power: no usable result within a reasonable time frame, prefer cohort analysis, qualitative research or a phased pilot.
• Non-isolable change or strong network effects: on a marketplace or a social graph, one arm contaminates the other, use a switchback or cluster design rather than a simple user split.
• Irreversible or sensitive stakes: safety, compliance or risk to the exposed user, decide by review and do not subject a live cohort to the experiment.

Description

An A/B test is a randomised controlled experiment. Live traffic is split at random between a control (A) and one or more variants (B), each unit of randomisation, user or session, seeing only one version. Because the assignment is random, the only systematic difference between the arms is the change under study, and a metric gap is attributable to it. This is why the technique is the online, randomised application of hypothesis testing: the statistics are the same, null and alternative hypothesis, significance threshold, power, p-value, confidence interval; what is specific to it is randomisation over real traffic. It thereby measures the real, in contrast to simulation, which predicts an outcome from a model without exposing any user.

Conduct begins before any data. The hypothesis is stated and the OEC (overall evaluation criterion) is chosen, the primary indicator measurable in the short term that the team believes carries the long-term objective, together with guardrails to monitor. The minimum detectable effect is then fixed, the smallest gap that would be worth shipping, along with the threshold α (often 0.05) and the power 1−β (often 0.80). From these parameters and the variance of the metric the sample size is computed, then the duration from the expected traffic. Under-sizing is decided here: a test that is too small cannot detect the target effect, and its null result teaches nothing.

Assignment happens at the right unit, and the realised split is checked by a sample ratio mismatch test: a split that departs from the intended ratio, 50/50 for instance, betrays a data or assignment defect that biases the result. It is to be diagnosed before reading anything at all. The experiment is then left to run over whole business cycles, full weeks that cover weekday and weekend, without stopping at the planned N mid-cycle. A test that is too short measures the reaction to novelty, curiosity in front of an unfamiliar look or rejection of the change, when the decision turns on the stabilised behaviour.

Reading is done with the interval: the gap and its confidence interval are reported against the declared α, because a point estimate alone hides the uncertainty. The most common trap is peeking, repeatedly looking at a fixed-horizon test and stopping it as soon as it turns significant, which inflates the false-positive rate well beyond the nominal α; the horizon is fixed in advance, or a sequential method designed for continuous monitoring is adopted. A win on the primary metric that silently degrades a guardrail, latency, revenue or retention, is a loss, just like the optimisation of a local indicator, clicks, at the expense of the OEC. Finally, slicing a null result after the fact until a segment turns out significant manufactures false positives through multiple comparisons: a segment finding is a hypothesis for the next test.

AI

AI serves A/B testing at its periphery. It generates candidates to test, copy, layouts or visuals, and feeds the pipeline with hypotheses. It speeds up analysis: extracting results, drafting the report, flagging anomalies. It automates the detection of quality problems, ratio-mismatch alerts, outliers, guardrail monitoring. And sequential or always-valid tooling makes it possible to follow a test continuously without the peeking penalty.

It does not replace judgement where judgement decides. It cannot fix the OEC: defining what better means is a choice of business value. It cannot invent the real reaction of users, since the experiment exists to measure it, and a model's prediction is not evidence, this is again the boundary with simulation. It cannot certify a significance it did not compute: a plausible summary that skips the statistics or that whitewashes a result obtained by peeking, is worth less than no analysis. Finally, the data of an A/B test are user-level behavioural logs: entrusting them to an external model creates an exposure with regard to data protection, its purpose and its minimisation within the meaning of the revised FADP, the Swiss Federal Act on Data Protection.

Examples

An online outdoor-equipment shop in French-speaking Switzerland tests a guest checkout (variant B) against its current mandatory-account funnel (A). The OEC is the checkout completion rate; the guardrails are the average basket in CHF and the payment error rate. Traffic is about 20'000 checkout sessions per week, randomised 50/50 by session. The duration is set at two full weeks, to cover whole business cycles, which brings the sample to 20'000 sessions per arm, beyond the minimum that power requires.

The design is fixed before collection.

At the end of the two weeks, the results decide.

The confidence interval of the gap excludes zero: the gap is significant at the declared α, and it exceeds the minimum detectable effect retained. The average basket stays within the noise, CHF 148 against CHF 146, so the completion gain is not paid for by a drop in value, and the sample ratio mismatch check confirms a realised split of 20'000 against 20'000, in line with the intended 50/50. The verdict is to ship B.

Visualisations

The technique calls for two forms of figure. Its mechanism is a flow: a population of live traffic, a random split, two arms each exposed to a version, an outcome measured per arm, a statistical comparison, a verdict. This is a geometry of boxes and arrows, drawn as a flow diagram and made the main illustration. Its deliverables, the pre-registered design and the result table, are made of rows and columns: the deliverable is the table itself, read back arm by arm. From the flow, the reader takes that the gap is caused, two parallel arms for a single change; from the result table, that the verdict rests on the interval excluding zero.

Cost

Tools

The sample size and power calculation is done with a spreadsheet or a dedicated power calculator, ahead of launch. Execution assumes an experimentation platform that assigns users, serves the variants and logs exposure, whether it rests on server-side feature flags or on a dedicated assignment service; offerings such as Optimizely, VWO or GrowthBook cover this need, the last as open source. Analysis relies on the web analytics tool that carries the metric and on a statistical environment, R or Python, for the confidence interval and the ratio check; sequential or always-valid test libraries avoid the penalty of continuous monitoring. In a Swiss context, the hosting of behavioural logs and their place of processing fall under data-protection obligations and weigh in the choice of a platform.

Sources

• Kohavi, R., Tang, D. and Xu, Y., Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, 2020: the practitioner reference for the method, which fixes the OEC, the metrics, the guardrails and the platform apparatus.
• Kohavi, R., Longbotham, R., Sommerfield, D. and Henne, R. M., Controlled experiments on the web: survey and practical guide, Data Mining and Knowledge Discovery, 18(1), 2009, pp. 140-181: the founding synthesis that establishes the method and its vocabulary, control and treatment, A/B and multivariate.
• Cohen, J., Statistical Power Analysis for the Behavioral Sciences, 2nd ed., Lawrence Erlbaum Associates, 1988: the statistical foundation of effect size, power and sample sizing.
• Fabijan, A. et al., Diagnosing Sample Ratio Mismatch in Online Controlled Experiments, KDD '19, 2019, pp. 2156-2164: the taxonomy and practical rules for diagnosing a non-compliant traffic split.
• Johari, R., Koomen, P., Pekelis, L. and Walsh, D., Peeking at A/B Tests: Why It Matters, and What To Do About It, KDD '17, 2017, pp. 1517-1525: the demonstration of the cost of peeking and the remedy of always-valid tests.