Learn when an A/B test result is credible. Interpret p-values, confidence intervals and power, avoid peeking errors, and decide what to ship.
Choose one primary metric, your significance threshold, the smallest effect worth acting on and the required power before the test begins. Estimate the necessary sample size and minimum runtime, then document the stopping rule. These choices prevent you from changing the decision criteria after seeing an attractive result.
For a conversion test, you usually need the number of visitors or eligible users and the number of conversions for each variant. For continuous metrics such as order value, you also need the mean, sample size and variability. Use comparable groups and exclude neither inconvenient users nor unusual days after reviewing the result.
The p-value is the probability of observing a difference at least as extreme as the one measured, assuming there is no real difference between the variants and the test assumptions hold. A p-value of 0.03 is not a 97% probability that the variant wins. It only measures how compatible the data is with the no-effect assumption.
A confidence interval shows the range of effect sizes supported by the data under the chosen method. If an interval for the difference includes zero, the result is generally not statistically significant at the matching level. More importantly, check whether the interval includes effects that are too small, harmful or commercially irrelevant.
The required sample depends on the baseline rate, the smallest effect you want to detect, the significance threshold and the target power. Higher power and smaller target effects require more observations. In ab testing, calculate the sample before launch; if traffic is insufficient, test a larger change or use a more frequent, decision-relevant metric.
A standard fixed-sample test assumes that you evaluate the result at the planned endpoint. Checking repeatedly and stopping as soon as p falls below the threshold creates more chances for random noise to look significant. Either follow the predefined sample and stopping rule or use a valid sequential testing method designed for continuous monitoring.
Ship only when the result is statistically credible, the confidence interval rules out unacceptable downside, and the likely effect is large enough to justify implementation cost and risk. Also check data quality, test duration, sample balance and metric side effects. Statistical significance supports a decision; it does not replace product or business judgment.