Statistical Significance

The probability that an observed difference between test groups would not have arisen by chance, conventionally requiring a p-value below 0.05. Advertising tests are routinely underpowered: small lifts on small budgets produce results that look good and mean nothing. Sample size should be calculated before the test starts.