Benchmark contamination detection starts with a definition
Benchmark contamination is the presence of evaluation items, or closely resembling text, in a model's training data, so a benchmark score partly reflects memorisation rather than capability. Every detection method infers that exposure indirectly: from the training corpus, from model outputs, or from score patterns across benchmarks.
What a detector can and cannot show
A detector's output is evidence of shared text or anomalous behaviour, not a measurement of how much a score was inflated. A positive result makes exposure plausible; a negative result means the method did not find it.
Training-data matching: verbatim n-gram overlap
N-gram overlap is the oldest and most widely described method. It searches training text for long exact matches with benchmark items and needs corpus access. According to arXiv 2311.04850 and arXiv 2410.08385, major developers including GPT-4, PaLM and Llama have used it. We did not check the developers' own reports.
According to arXiv 2407.08716, n-gram tools are costly, assume access to pre-training data, and generally detect contamination only when clusters of test items co-occur.
Measurement choices that vary between labs
There is no uniform n-gram measurement. The value of n, fuzzy matching, skip-grams and the hit threshold all vary, as arXiv 2410.08385 notes. Two labs reporting clean results may have run different tests, so their numbers are hard to compare. The same paper distinguishes contamination types, such as question leakage, quoted news and common phrases, which behave differently; common-phrase overlap can give false positives.
Reported thresholds and their provenance
Two secondary survey descriptions, arXiv 2406.04244v1 and arXiv 2406.14644v1, attribute thresholds to developers: a 13-gram signal in GPT-3 work, a 50-character overlap in GPT-4 (reportedly) and a 10-token match in Llama 2. We did not open the original model reports, so treat these as unverified. No uniform threshold is agreed.
Canary strings: a benchmark-side defence
Benchmark maintainers embed a unique marker, a canary string, in benchmark files. If a model reproduces the marker, it probably saw the file. A missing marker proves nothing about paraphrased or translated use. Our sources for canaries are blogs and wikis (capitalandcompute.net and artificial-intelligence-wiki.com), used for orientation only. BIG-bench is cited there as an example, but we did not confirm it against the repository.
Where training-data matching breaks down
Rephrasing and translation
arXiv 2311.04850 argues, with experiments, that rephrasing or translating test data bypasses n-gram decontamination, so a model trained on rephrased items could score well while passing string checks. The same paper reports that embedding similarity struggles to separate rephrased questions from other questions in the same subject. The extent of this effect on current training pipelines is not established.
Synthetic data
Synthetic training data can resemble benchmark items in ways n-gram overlap does not flag. arXiv 2311.04850 cites the Phi-1 report, in relation to HumanEval, as an example. We have not checked that report.
Adversarial rephrasing against detectors
arXiv 2402.02823 reports that rephrasing test data with GPT-4, then making further changes to evade a detector, defeated the detectors tested while still raising benchmark scores by up to 15%. The authors say this applies to all detection methods current at the time. These are their own results; we found no independent replication.
Behaviour-based methods: membership inference and perplexity
These methods need only model outputs, not the corpus. Perplexity tests assume contaminated text is less surprising to the model. Membership inference asks whether a given example was in training. arXiv 2406.04244v1 reports criticism of membership inference, and arXiv 2402.02823 reports that it can be evaded. Generic text can also be unsurprising for reasons unrelated to memorisation (the author's reading).
A 2026 preprint, arXiv 2603.03203, proposes an output-distribution method for small models, using perplexity baselines for comparison. We did not assess its results in detail.
Performance-based detection and the reference benchmark
ConStat (arXiv 2405.16281, published at NeurIPS 2024) defines contamination as inflated performance that does not generalise, not as the presence of benchmark text. It compares a primary benchmark with a reference benchmark for the same task, and needs no training data. The authors report high contamination in several popular open models; these headline results are self-reported on their own benchmarks.
It detects inflation but cannot identify its cause, so a score gap could in principle come from something other than contamination. Results depend on the reference benchmark chosen. The work covers open models and predates the reasoning models now under evaluation.
Filtering, reasoning models and open questions
Filtering and its effect on scores
Removing flagged documents does not reliably remove contamination's effect on scores. A summary on alphaXiv of arXiv 2401.06059 reports that n-gram filtering removed more than 30% of tokens in some cases, with little change in performance. We saw only that summary; the figure is unverified against the paper.
Reasoning models and closed-model coverage
arXiv 2510.02386, a preprint, argues that chain-of-thought training and reinforcement learning may conceal evidence that memorisation tests would otherwise catch. We read only its abstract. arXiv 2605.21856 discusses evasive contamination and a zero-CoT truncation method, but we found no validated results. Most methods need model weights, training data or repeated black-box queries, so closed-model coverage is thin.
Some benchmarks avoid detection altogether through post-cutoff items, rotating test sets or contamination-resistant release. MMLU-CF and AntiLeak-Bench are examples, listed on emergentmind.com. Whether they hold up against modern training pipelines is not yet established by independent evidence.
Reading a detection result on a model claim
A positive n-gram match shows shared text, not that the model learned from it or that its score changed. A negative result does not prove absence. Before weighing a claim, ask which method was used and with what n, matching rules and threshold, whether a reference benchmark was used, and who ran the test.
Whether a score gap is meaningful is a separate step; see statistical power when comparing models and benchmark saturation evidence.