Releasing model weights often does not, by itself, let a reader regenerate a reported LLM benchmark score. The evaluation script, inference code, decoding configuration and software stack can each change the number, and weights files contain none of them. The clearest written statement of this is the Model Openness Framework (MOF), a proposed standard that treats weights-only releases as incomplete. What the evidence does not establish is how often these gaps actually change published results.
That distinction matters now because many labs publish weights and a summary table, and readers treat the table as checkable. This article sets out which parts of the problem are documented, which rest on single studies, and what a careful reader can do with a reported score today.
Terms used in this article
Reproduction here means regenerating a reported result from the same data, code and artifacts. The Vector Institute's explainer on machine learning for healthcare uses reproducibility in this same-data, same-code sense, and notes that reproducible results may still fail to replicate or generalise. The MOF uses "completeness" for something different: whether the full pipeline is shared in usable form. The two definitions differ in scope, so a claim that rests on one should not be read as a claim about the other. Replication means reaching a comparable result with different data, code or implementation. Open weights means the trained parameters are downloadable and says nothing about the code that trained, evaluated or served the model.
What a weights-only release leaves out
The MOF lists the components it treats as relevant to reproducibility: training, validation and test code, inference code, evaluation code, evaluation data and supporting libraries. Its authors state that evaluation results often cannot be reproduced for lack of disclosure. That is the authors' characterisation, not an independent measurement.
The framework sorts releases into classes by how much they disclose. The class the MOF calls "open model", meaning weights with little else, is the one it places below completeness. The framework's class tables were not checked in full for this article, so the boundaries between the other classes are not described here. The practical test it implies is simpler than any classification: can a third party run the evaluation end to end from what was published?
A release can be open in the licensing sense and still fail that test. Hosted API access is a weaker form of access again, because the served model can change without a version change visible to the user. The full text of the NeurIPS checklist language that some arXiv papers paraphrase on hosted access was not retrieved, so no wording from it is quoted here.
Evaluation and inference code
The MOF requires that inference code include whatever is needed to replicate benchmark results, and that evaluation code be released under an open-source licence. No source found measures how often labs meet either requirement.
A benchmark score depends on the prompt template, answer extraction, decoding parameters, few-shot example selection and scoring script. A weights file contains none of these. In principle, two groups running the same checkpoint with different harness versions could report different scores without either making an error. No source retrieved for this article quantifies that effect for a specific harness, which is why the mechanism is stated here as a possibility rather than a measured rate.
The inference engine adds a further layer. A related-work section of a 2026 paper, "The Silent Hyperparameter", characterises the inference engine as an under-documented source of evaluation variance. Only that section was retrieved, so the paper's own results are not reported here.
Numerical nondeterminism and the inference stack
A preprint (arXiv 2506.09501) reports that greedy decoding can produce different outputs across runs because of numerical precision, and attributes this to non-associative floating-point arithmetic. Floating-point addition is not associative, so summing the same numbers in a different order can round differently. Parallel GPU kernels and batch-dependent scheduling can change that order. The retrieved abstract supports the attribution at this level; the specific kernel and batch conditions the authors tested were not confirmed from the material available, and the finding's reach across hardware is not established.
A NeurIPS 2025 poster (virtual page 118169) reports that DeepSeek-R1-Distill-Qwen-7B varied by up to 9% in accuracy and by up to 9,000 tokens in response length when GPU count, GPU type and evaluation batch size changed. The benchmark behind these figures is not named in the material retrieved, so it cannot be identified here. The poster is a summary, covers one model and one setup, and should not be read as a typical effect size.
A February 2026 paper, "Same Prompt, Different Outcomes", summarised on alphaXiv, reports non-identical outputs across repeated runs even at temperature 0. That summary is an aggregator page, not the paper itself, and its tasks are data analysis rather than benchmark scoring.
Data, training and the frontier-scale gap
The clearest measurements of accessibility concern dataset and code availability rather than reproduction. A study of 511 machine-learning papers, published in Science Translational Medicine, found that machine learning for health compared poorly with other subfields on such metrics. The sample is older, and the study's abstract on PubMed was the only text retrieved, so it is a dated signal about accessibility, not a current baseline.
A 2024 Data Science Journal paper, as indexed by its DOI, describes a bronze-level standard that includes data, trained models and source code, and states that access to code and data does not guarantee reproduction. It is a single applied case. For frontier-scale models, training-data reproducibility is largely unaddressed by the sources found.
Where a weights-only release is enough
Weights do let others reproduce a result without retraining, which matters when training is expensive. A checkpoint with a fixed evaluation script supports a comparison that a bare score cannot. The position that follows from the evidence is narrower than either extreme. Weights are sufficient for reproducing a reported number when the evaluation harness, decoding configuration and software stack are also fixed and recorded, and that record is what most releases lack.
Established, contested and open
- Established as a framework position: the MOF's list of reproducibility-relevant components and its requirements for inference and evaluation code. These are the proposal's own terms, and the framework is not a measurement of practice.
- Reported by one team, not independently replicated in the material reviewed: greedy-decoding variability from floating-point non-associativity (arXiv 2506.09501), and the up-to 9% accuracy swing for one model in a poster summary.
- Contested or unsettled: whether greedy decoding is deterministic across software and hardware stacks. The reported evidence points towards no, but rests on a small number of experiments.
- Open: the share of LLM papers that fail to reproduce under each openness class, and whether MOF completeness correlates with successful reproduction. No source found measures either.
Practical implications
When a paper releases weights, look for the evaluation script, the inference code, the exact decoding configuration and the library versions, and record the GPU type, GPU count and batch size of any run you repeat. Treat a single greedy-decoding score as one sample. Where budget allows, run the evaluation several times on the same hardware and report the spread.
The spread is what decides whether a claimed gain means anything. Consider a paper reporting a 2-point improvement over a baseline, where repeated runs on a fixed setup vary by 9 points. The 2-point gap sits well inside the noise, and the comparison supports no conclusion until the spread is measured for both models under identical conditions. Benchmark saturation and contamination make the same point from another direction: a small reported difference on a near-saturated or possibly leaked benchmark carries less information than its precision suggests. Statistical power, meaning how many items and how many runs a comparison rests on, determines whether a difference of a point or two can be distinguished from run-to-run noise at all.
Avoid citing a hosted API as reproducible access unless the model version and serving date are pinned.
Limitations and open questions
The sources reviewed are preprints, a conference poster summary, a framework proposal, an aggregator page and abstracts. Few of them measure reproduction rates directly, and the nondeterminism figures come from single teams and single models. The corpus-level study that might address rates could not be checked, and its title and record did not match, so it is not used. A settling study would take published LLM results, release different subsets of artifacts, and measure how often each subset regenerates the reported number under stated hardware and software conditions.
Evidence assessment
The sections above set out what each source supports. This is the same judgement, sorted by strength.
Strong: the MOF's list of reproducibility-relevant components, as a description of what the framework names. Open weights alone do not define a complete, runnable evaluation pipeline.
Moderate: the MOF authors' statement that evaluation results often cannot be reproduced for lack of disclosure, which is the framework authors' characterisation; greedy decoding can vary across runs because of floating-point non-associativity, as one team reports; code and data availability do not guarantee reproduction, per a single applied workflow paper.
Weak or preliminary: the up-to 9% accuracy and 9,000-token variation for one model in a poster summary; the non-identical temperature-0 outputs in data-analysis tasks, known only from an aggregator; the inference engine as an under-documented source of variance, from a related-work section alone; the self-reported regeneration claim in the Numbat preprint, which is unaudited.
Unknown: the share of LLM papers that fail to reproduce under each openness class; whether MOF completeness correlates with reproduction success; training-data reproducibility for frontier-scale models; the size of harness, quantisation and serving-engine effects across models and benchmarks beyond the single poster.
Editor's take
The evidence supports one narrow judgement with reasonable confidence: an open-weights release is not a reliable stand-in for a reproducible result, because the settings that fix a reported number usually live outside the weights file. The MOF's component list is a sensible checklist, and nothing retrieved contradicts it. The evidence does not say how often this matters in practice, and calling the failure rate high or low would go beyond what has been measured.
The practical reading is that reproducibility belongs to a complete run specification, not to a model artifact. A lab that publishes weights without its harness is not necessarily acting in bad faith, but a reader should hold the reported number as unverified until someone has run the published evaluation. Where the claimed difference is smaller than the run-to-run spread, the question has become a statistical one as much as a software one.