ab-test-analysis

A/B Test Analysis - ab-test-analysis

Skill Overview

Use statistically rigorous methods to evaluate A/B test results, provide significance tests, sample size validation, confidence interval calculations, and recommendations on whether to launch, extend, or stop an experiment.

Applicable Scenarios

1. Evaluating Experiment Results and Determining Statistical Significance

After completing an A/B test, you need to objectively determine whether the difference between the treatment and control groups is real or merely caused by random variation. This skill helps you calculate p-values and confidence intervals and provides a clear conclusion on whether the result is statistically significant, helping you avoid making decisions based on intuition or luck.

2. Validating the Soundness of the Experimental Design

Before reviewing the results, first check whether the experiment ran long enough, whether the sample size was sufficient, and whether there were issues such as sample ratio mismatch. This skill evaluates statistical power using statistical formulas and identifies whether the test was inadequately designed, such as having power below 80%, preventing unreliable conclusions caused by flaws in the experiment itself.

3. Interpreting Experiment Data and Making Product Decisions

When dealing with data from multiple experiment groups and multiple metrics—including primary and guardrail metrics—you need to make a comprehensive judgment about whether to roll out the change, continue observing, or abandon the launch. This skill not only calculates statistical metrics but also provides clear Ship / Extend / Stop / Investigate recommendations based on business significance, along with explanations and next steps.

Core Functions

1. Statistical Significance Testing

Calculate the conversion rates of the control and treatment groups, relative lift, p-values using a two-tailed z-test or chi-square test, and 95% confidence intervals to clearly answer the key question: “Is the difference statistically significant?” If you provide a raw data file (CSV, Excel, or an export from an analytics platform), this skill can generate and run a Python script for precise calculations without relying on manual estimates.

2. Sample Size and Experimental Design Validation

Use the statistical formula n = (Z²α/2 × 2 × p × (1-p)) / MDE² to assess whether the current sample size is sufficient to detect the expected effect and calculate whether statistical power meets the 80% standard. Also check whether the experiment duration covers 1–2 complete business cycles, whether there is sample ratio mismatch (SRM), and whether the novelty effect has faded, ensuring the validity of the experimental conclusions.

3. Comprehensive Decision Recommendations

Based on statistical significance, practical business impact, and guardrail metric performance, provide clear Ship / Extend / Stop / Investigate recommendations, along with explanations and follow-up actions. The goal is not only to answer “Is there a statistically significant difference?” but also “Should we launch it?”—helping product teams make data-driven product decisions.

Frequently Asked Questions

How large a sample size is needed for a reliable A/B test?

Sample size depends on the minimum detectable effect (MDE) you want to identify, the current conversion rate, and the statistical significance level. Use the formula n = (Z²α/2 × 2 × p × (1-p)) / MDE² to calculate it and ensure that statistical power exceeds 80%. If the sample size is insufficient, even a “positive trend” may simply be random variation. In that case, you should extend the test or accept that no definitive conclusion can be drawn.

Can we launch as soon as the p-value is less than 0.05?

Not necessarily. A p-value below 0.05 indicates that the difference is statistically significant, but you also need to check whether the magnitude of the improvement has meaningful business value—for example, a conversion rate increasing from 5% to 5.1% may be statistically significant but not worth launching—whether guardrail metrics such as revenue, engagement, and load time have deteriorated, and whether the experiment ran long enough to eliminate the novelty effect. This skill considers all these factors and provides recommendations such as Ship or Investigate.

What should we do if the experiment shows no significant difference but the trend is positive?

If p > 0.05 but the direction is favorable, the sample size may be insufficient or the true effect may be small. In this case, do not launch immediately. Consider extending the test to collect more data, or accept that the current effect is too small to justify further investment. If the trend is flat, with no clear direction, stop the test to conserve resources. This skill provides clear Extend or Stop recommendations based on power analysis and the direction of the trend.