AI benchmark saturation, as a 2026 preprint defines it, means a benchmark has lost reliable discriminative power among state-of-the-art models: top systems score so close together that the test no longer separates them. That describes the measurement, not whether the underlying task is solved. This article separates what the evidence establishes from what is contested or still open.
What 'saturated' means and how it is measured
Working definition and threshold
The definition comes from a 2026 systematic study of LLM benchmarks (arXiv:2602.16763). The study also proposes an uncertainty-aware index built from leaderboard data. Both the threshold and the index are its own choices, so a 'saturated' label means what that study's threshold says and no more.
Ceiling clustering versus a solved task
Scores bunching near the top are the visible symptom. The study cautions that strong clustering at a low score should not be read as the task being solved. A tight cluster can reflect a test that cannot separate models, ambiguous items or wrong answer keys, so the uncertainty around each score matters as much as the ranking.
How widespread AI benchmark saturation is
The 2026 study of 60 LLM benchmarks
The study analysed 60 benchmarks across 14 properties (ICML 2026 poster abstract, icml.cc/virtual/2026/poster/63311) and found nearly half saturated under its threshold. Associations reported from this one preprint: saturation rose with benchmark age and scale; private hidden test sets gave no protective effect; expert-curated benchmarks resisted saturation better than crowdsourced ones. The 'nearly half' figure describes these 60 benchmarks under this threshold. It is not an estimate for the field.
Earlier evidence from 3,765 benchmarks (2022)
A 2022 Nature Communications study curated 3,765 computer vision and NLP benchmarks. It found a large fraction trending toward near-saturation quickly and many benchmarks never gaining wide use, and it recommended emphasising versatility, breadth and real-world utility. It predates LLMs, so it supports the pattern but not the LLM-era timeline.
Competing explanations for saturation
Models improving
The simplest account is that models get good enough to fill the test. Coverage of the Stanford AI Index 2026 reports SWE-bench Verified scores rising from about 60% to nearly 100% in a year (c3.unu.edu). The report itself was not retrieved, so this is secondary.
Benchmark-side limits
A bio-benchmark preprint (arXiv:2505.06108) reports plateaus around 75–90% on PubMedQA, MMLU-Bio and WMDP-Bio, while the same models keep improving on harder tests. The authors argue the plateaus reflect benchmark limits such as ambiguous or wrong answer keys. This is one preprint's argument, not a measured error rate.
Contamination
An ARC Prize observation, as summarised by DataCamp, suggests some frontier models may have seen ARC data in training: a Gemini 3 reasoning trace referenced ARC-specific colour mappings unprompted. The primary ARC Prize post was not read. The size of contamination's effect is not quantified in the sources reviewed.
Resolution
A 2026 SEAL preprint (arXiv:2605.30104) argues top systems can tie on a task-level outcome yet differ in quality, and that a changed judging protocol recovered ranking signal on saturated benchmarks. The results are on the preprint's own setup.
Does saturation mean progress has stalled?
This is contested. The bio-benchmark argument and AI Index coverage both point against stalling, but the AI Index claims are secondary. Coverage says evaluations meant to stay challenging for years are saturating in months (splunk.com, quoting the Index). The material reviewed contains no dated score histories for the same benchmark across labs, so acceleration cannot be tested. The author's reading is that saturation mainly shows which tests remain informative, and is weaker evidence about the frontier trend itself.
What has replaced saturated benchmarks
The pattern is serial: harder static exams, live question pools, interactive tests, then proprietary evaluations. Whether replacements saturate faster is not yet measured. A 2026 survey (arXiv:2606.30219) says these designs cannot be placed on one difficulty scale, that replacement carries curation, adoption and standardisation costs, and it explicitly does not estimate a replacement half-life.
Humanity's Last Exam
The HLE authors say LLMs scored above 90% on popular benchmarks such as MMLU, which limited measurement. HLE was built as a closed-ended academic benchmark at the frontier of human knowledge (arXiv:2501.14249). It was published in Nature (vol. 649, pp. 1139–1146, 2026, per agi.safe.ai). The working paper says it 'may be the last academic exam we need to give to models, but it is far from the last benchmark' (ETH Zurich copy, read from a PDF extraction).
ARC-AGI-1 to ARC-AGI-3
ARC-AGI-1 showed signs of saturation near human-level performance, so ARC-AGI-2 added a new task set and human calibration (arXiv:2505.11831). ARC-AGI-3 uses interactive environments where rules emerge from the agent's own actions (DataCamp tutorial, secondary). One secondary source reports frontier systems below 1% on ARC-AGI-3 at launch in March 2026. I could not check this against official figures.
Proprietary and weighted approaches
Vendor material describes labs running private pipelines with fresh production tasks and held-out data. The imerit post names no labs or pipelines, so these claims are unverifiable here. A preprint (arXiv:2607.01740) proposes weighting benchmarks automatically by how well they separate top scorers in a rolling window. It was not validated in the material read.
Reading HLE and ARC-AGI scores
There is no single HLE score. Figures differ by tracker, subset, tool-use rule and date:
- BenchLM HLE page, updated 17 October 2026: 65% across 76 models (benchlm.ai/benchmarks/hle). Protocol not confirmed.
- llm-stats: 65.0% across 108 models (llm-stats.com).
- Artificial Analysis, as cited by Digitortoise: 59.1% on a 2,158-question text-only set (digitortoise.com).
- Scale AI official text-only board, as reported by BenchLM: 54.2%, dated 12 October 2026.
- Scale AI public-question leaderboard: 46.44% as of 16 August 2026, per IntuitionLabs.
For ARC-AGI-2, a directory site reports about 54% in December 2025 and 92.5% by July 2026 on the semi-private set, while BenchLM reports 95% in October 2026. These are low-confidence aggregators that conflict, and the model names attached to them could not be verified.
Cite protocol, subset, date and source with every number. A bare 'HLE score' is not citable.
What remains open
- Replacement half-lives: not estimated by the survey, and no dated score histories were found.
- Whether replacements saturate more slowly: not measured.
- Selective reporting: how labs choose which benchmarks to report. I found almost no primary sources on this.
- Whether saturation is accelerating.
- The size of contamination's effect.