A one-point gain on a benchmark means little until you know how many test items produced it and how much the two models' scores vary item by item. Statistical power tells you whether the comparison could have detected that gain at all. This reference covers paired comparisons of two model variants on a shared test set, and it separates what the evidence supports from what is contested or open.
What power means for a model comparison
In a comparison of two variants, the null hypothesis is that they perform equally on the test distribution. A type I error is rejecting a true null, declaring a difference that is not there. A type II error is failing to reject a false null. Power is one minus the type II error rate: the probability that the test detects a real difference of a stated size. The minimum detectable effect (MDE) is the smallest true difference that a given design detects with a stated power at a stated significance level.
Power is therefore a design-time quantity. It is computed from an assumed effect size, an estimated variance of the paired per-item differences, the test-set size and the chosen alpha. None of these is known exactly before the experiment, so any power figure is only as good as its inputs. A reported gain does not carry a power value with it; power belongs to the design.
Why small test sets leave comparisons underpowered
Card et al. (EMNLP 2020) make this case for NLP benchmarks. They report that for several GLUE tasks, small test sets mean most attempted comparisons to state-of-the-art models are not adequately powered. They also argue that power is often ignored in NLP comparisons. They frame results in terms of the MDE: with a small test set the MDE can be large, so only large improvements yield comparisons with at least 80% power.
The underlying mechanism is standard. For a fixed per-item variance, the standard error of the mean paired difference falls roughly as the inverse square root of the number of items. Detecting an effect half as large therefore needs roughly four times as many items. This is a scaling rule, not a sample-size formula for any particular benchmark.
Estimating the inputs to a power calculation
Card et al. describe three sources for the parameters a power calculation needs: development data, historical regression of results, or assumptions where neither is available. The outputs are design-time estimates. They describe what a planned test set could detect given those inputs, not what a finished comparison actually detected.
Variance matters as much as the accuracy gap. Two pairs of models with the same mean difference can disagree on very different sets of items, and the variance of the paired differences is what enters the calculation. Variation from retraining with different random seeds and variation from sampling test items are different sources of uncertainty. The sources I found do not give a clean method for separating them. I treat that as an open gap, not a solved step.
Choosing a test and what it does to error rates
No single test is universally correct, and test choice affects both type I error and power. Dietterich (1998) reports that two widely used statistical tests have a high probability of type I error in certain situations and should not be used. The same study reports the cross-validated t test as the most powerful of the tests compared. The abstract describes the 10-fold CV t test's type I error as somewhat elevated rather than high. A secondary tutorial describes this test differently. I have not resolved that conflict against the full paper, so cite the primary abstract and check the full text before relying on either description.
Comparing across multiple datasets
Demšar (2006) recommends the Wilcoxon signed-ranks test for comparing two classifiers. For more than two classifiers over multiple datasets, it recommends the Friedman test with corresponding post-hoc tests. I read these recommendations from the Semantic Scholar abstract page. I did not retrieve the JMLR original.
Reading a non-significant result
Failing to reject the null is not evidence that two models are equal. A low-powered comparison often fails to reject even when a real difference exists, so a non-significant result can mean the test could not see the difference. Benavoli and colleagues report that null-hypothesis significance testing fails to reject the null in 35 of 54 datasets, and that their Bayesian analysis declares the two classifiers equivalent in 6 of those. This is one paper's analysis. The source wording does not make clear which datasets the 6 refers to, so check the paper before quoting the count.
Equivalence needs its own framing. A test that asks whether a difference lies inside a margin fixed before the experiment answers a different question from one that asks whether the difference is zero. The Bayesian and frequentist positions remain contested. The Bayesian side argues that frequentist tests have shortcomings that Bayesian hypothesis testing can overcome. Frequentist tests are still widely used, and Bayesian methods are an alternative, not a settled replacement.
Statistical significance is not practical significance
A large test set can make a very small difference statistically significant. Significance answers whether a difference is distinguishable from zero given its variance. It does not say whether the difference matters for the application. Report the effect size with an interval, and judge its relevance against a threshold written down before the experiment. The principle is standard statistics, but the only direct source I found is a forum answer about regression. The ML-specific application therefore rests on the primary papers above.
Open questions
- Power for LLM and generative benchmarks, where outputs are not a single accuracy score, received little coverage in the sources I found.
- No settled method for separating seed variance from item variance appeared in the sources I found.
- A 2026 LREC paper introduces a toolkit for simulating evaluation scenarios and estimating MDEs. I saw only its abstract, so I have not checked its methods or results.
A workflow before the next run
- State the effect size you need to detect before looking at results, and explain why that size matters for your application.
- Estimate the variance of paired per-item differences from development data or a pilot run, and record how you estimated it.
- Work out the test-set size that gives your chosen power at your chosen alpha, or report the MDE your existing test set supports.
- Fix the test before seeing results, and report its name and assumptions.
- Report the effect size with an interval, and state whether it clears your pre-set threshold for practical relevance.
If a comparison fails the third step, the honest report is that the test set cannot resolve the gain you care about, not that the models are equal. Whether a benchmark still discriminates between models at all is a separate question; see AI benchmark saturation.