Annotators often pick different winners for the same pair of model responses, and in the datasets that have been measured closely, the split is large enough to matter. The open question is what the split means. It could be noise that a better pipeline should remove, or it could be real variation in what people want from a response that a single reward score erases. The recent literature, almost all of it arXiv preprints, supports the first observation more firmly than the second.
Zhang et al. (arXiv 2410.14632) report that over 30% of examples in two datasets, MultiPref and HelpSteer2, showed diverging preferences. That is a measured rate for two datasets, not a general property of preference data. The same work points to response style and task underspecification as major drivers. Position papers from 2026 argue that much disagreement reflects annotators applying different constructs under one label, and they propose consistency checks to test that. Those are arguments, and the empirical support for them is thinner.
This article sets out the proposed causes, what each rests on, and what a dataset team can reasonably do now. It does not cover how agreement statistics are computed; for that, see what inter-annotator agreement measures and misses.
Terms and scope
A pairwise preference label records which of two model responses to the same prompt an annotator judges better under a stated criterion. The Bradley-Terry model fits one latent score per response so that the probability that response A beats response B depends only on the difference between their scores; reward models are commonly trained on this objective. Inter-annotator agreement is the rate at which annotators give the same label, and chance-corrected versions such as kappa adjust for agreement expected by chance.
How common disagreement is
The most direct primary figure comes from Zhang et al. (arXiv 2410.14632), who report that over 30% of examples in MultiPref and HelpSteer2 showed diverging preferences. The figure applies to those two datasets. No cross-dataset rate is established in the sources reviewed here. The finding rests on one preprint, read at abstract and introduction level, so it should be treated as well supported for these datasets and not yet as a general rate.
Criterion ambiguity and measurement non-invariance
Consider the label "helpful". One annotator may read it as informative, another as emotionally supportive, and a third as something they can act on. A position paper (arXiv 2604.03238) argues that annotators in such cases apply different constructs under one label, which it calls measurement non-invariance, rather than holding different preferences about the same thing. The paper is a conceptual argument with illustrative examples, not an experiment.
The three measurement conditions
The paper says the standard measurement model for preference labels holds only if three conditions are met: the constructs are well defined, annotator attitudes are stable, and the elicitation is valid. It proposes test-retest, framing and order consistency checks to test them. Whether those checks would separate construct ambiguity from ordinary annotator error is not shown in the sources reviewed.
Interpreting the PRISM classification
An earlier version of the same paper (arXiv 2604.03238v1) reports a classification of flagged inconsistencies in PRISM data, with raw counts of 25 of 44 as non-attitudes and 14 of 44 as constructed preferences. The sample is small and the categories are interpretive. Use the counts as an illustration of the argument, not as a prevalence estimate. The counts have not been checked against the paper's full text.
Response style, length and underspecified tasks
Zhang et al. name response style and task underspecification as major drivers of divergence. A detailed answer may please one user and read as padded to another. A cautious answer may suit a safety-critical setting and frustrate someone asking a casual question. These examples are illustrations from a secondary explanatory chapter (arXiv 2608.00133); no effect size for either driver was found in the sources reviewed.
Instruction framing changes agreement
The clearest measured effect of framing comes from outside preference data. In a Chinese noun-phrase annotation study (arXiv 2403.04376), removing task framing lowered kappa for definiteness from 0.67 to 0.48, while agreement on plurality held. The result shows that wording can move agreement; it gives no effect size for preference labels. A law-domain paper (ACL Anthology 2022.law-1.6) reports that annotator labels differ across groups and that task-definition framing matters. Only fragments were available, so the specifics are not given here.
Annotator background, culture and values
Hidden Consensus (arXiv 2606.10569) argues that pooled scalar rewards can collapse several acceptable answers into one score. It cites the PRISM dataset, as reported by Kirk et al. (2024), as covering 1,500 participants from 75 countries. Those figures are as cited by arXiv 2606.10569, not checked against Kirk et al. directly.
Majority voting and minority values
A study on the HateXplain dataset (arXiv 2606.28772) hypothesises that majority voting may silence minority annotator values at the boundary between hateful and merely offensive content. The paper reports findings on one dataset with one model family, and the causal link to majority voting is not established. Read it as a hypothesis worth testing, not a result.
Expert versus crowd annotators
A reward-robust RLHF paper (arXiv 2409.15360) compares expert internal annotators with external general-public annotators. Results were not retrieved, so no direction or size for any gap is given here. A content-moderation literature summary (arXiv 2604.16654) reports that prior work finds crowd and expert annotators differ systematically; that is a secondary summary.
Does aggregation change what the model learns?
This question is contested. Common options are simple averaging, majority vote, dropping annotators with high inconsistency, and pooling everything into one scalar reward. A secondary review hosted at pith.science reports an 18.6% label-flip figure from filtering high-inconsistency annotators, attributed to arXiv 2604.03238. The review itself disputes the basis of that figure, and the primary paper was not checked, so the number should not be repeated as established. No evidence retrieved shows that filtered labels track true preferences more closely than unfiltered ones.
Systematic disagreement versus noise
Krippendorff's distinction, as cited in arXiv 2110.05719, separates unavoidable random variation from systematic disagreement driven by data or annotator features. One way to frame the question for a dataset is to ask whether disagreement concentrates on an identifiable feature, such as prompt type, annotator group or response length, or whether it looks like scatter. A vendor guide from Toloka says low agreement often signals ambiguous prompts, unclear instructions or annotator fatigue. That is a practitioner view, not evidence.
Two mitigations are proposed. Reliability-aware training (arXiv 2607.20515) estimates annotator reliability and weights labels accordingly; this description comes from the abstract only. Selective LLM evaluation, which abstains when confidence is low, is reported in an ICLR 2025 paper; I could not pin down a locator for that paper beyond the proceedings URL, so the claim should be checked before use. A QA study (arXiv 2408.09235) reports that LLM judges may show lower and more variable agreement than human annotators in some settings.
Evidence assessment
The evidence is strongest on the descriptive point that disagreement is common in some widely used preference datasets. It is moderate on the claim that disagreement tracks identifiable features, and weak on which causes dominate. Most sources are preprints of unclear peer-review status, and the full texts of most were not checked. No peer-reviewed meta-analysis of preference disagreement sources was found.
Practical implications
Dataset teams should record the criterion and instructions alongside every label, because the sources that test framing show that wording can move agreement. Keep per-annotator labels so disagreement can be broken down by annotator group and prompt type before any aggregation is chosen. Compare aggregation options on held-out annotators rather than assuming that the one producing the cleanest training signal is the most accurate. Avoid treating high agreement as proof of correct labels, and avoid treating a scalar reward as a neutral summary of what annotators wanted. This is the author's reading of the evidence, not a finding of the sources.
Limitations and open questions
Cross-dataset disagreement rates are unknown. The causal roles of response style, underspecification and annotator background have not been measured together in one controlled design. Expert versus crowd differences are unresolved in the sources reviewed. Whether filtering improves a model against held-out human judgements has not been settled. A controlled study that varies task wording, instructions, annotator pool and presentation together, and then scores models against held-out annotators, would address most of these gaps.
Editor's take
The reasonable judgement is that disagreement in preference data is a measured feature of some datasets, and that its interpretation is still open. The best-supported evidence is the Zhang et al. rate for two datasets and the framing result in a non-preference task. The argument that disagreement is construct variation rather than error is plausible and worth testing, but it currently rests on position papers and illustrations.
That leaves dataset teams in an awkward spot. Treating all disagreement as noise to be filtered throws away information that may be real, and the filtering evidence has not shown gains. Treating all disagreement as legitimate pluralism ignores that wording and instructions can produce disagreement that no annotator actually holds. The practical course is to measure where disagreement sits, document the criterion, and decide aggregation on evidence from held-out annotators.
"