Inter-annotator agreement (IAA) is the degree to which two or more annotators assign the same labels to the same items. It measures consistency among annotators. It does not measure whether any label is correct, and the two are often conflated. The simplest quantity is observed agreement, p_o: the proportion of items on which the labels match. It is easy to compute and easy to misread.
What the Number Measures and Why Raw Agreement Is Not Enough
Krippendorff's 2004 article in Human Communication Research argues that percent agreement is problematic partly because extreme disagreement is statistically unexpected, so raw agreement cannot by itself interpret departures from perfect agreement. I read this through a Scribd copy, not the journal page. An illustrative case from an Arize blog post makes the point concretely: one annotator flags 5 failures in 100 items, and a second labels every item “pass”. Percent agreement is 95%, yet the second annotator never identified a failure. This is a blog illustration, not a study.
How Chance Correction Works
Cohen's kappa corrects p_o for the agreement two raters would reach by chance:
κ = (p_o − p_e) / (1 − p_e)
Here p_e is computed from each rater's own marginal label distribution, the proportion of items each rater assigns to each category, multiplied across raters and summed over categories. The formula and its definitions are as given in arXiv 2206.05446.
κ = 0 means observed agreement equals what the marginal distributions predict by chance. It is not a quality level. A κ of 0.6 on one dataset and 0.6 on another may reflect very different prevalence structures.
Chance is not a fixed quantity. Artstein's handbook chapter, as copied by DTIC, argues that chance agreement reflects what arbitrary annotation would produce, and that arbitrary annotations can be highly skewed and yield high chance agreement. Its hypothetical example, as the research summary describes it, shows reliability only 5% and 18% above chance within separate subsets but almost 70% when the subsets are pooled. Artstein treats this as a warning about pooling. The numbers are hypothetical, not empirical results.
Choosing a Coefficient for Your Design
Cohen's kappa is defined for two raters and categorical labels. Fleiss' kappa extends to more raters, but, according to a vendor comparison table from Appen, it requires a fixed number of raters per item. Krippendorff's alpha accommodates any number of raters, raters varying across items, missing ratings, and nominal through ratio measurement levels. A University of Pennsylvania-hosted paper quotes Krippendorff's 2004 position that alpha is general across number of observers, measurement levels, sample sizes and missing data. The Appen table is vendor material and should be checked against the original papers before you cite it.
Two raters versus many
With exactly two raters and a complete design, kappa is the conventional choice. Once raters vary by item, or the number of raters grows beyond two, the coefficient choice matters more, and the design should drive it rather than habit.
Missing data and measurement level
Missing ratings and ordinal or interval labels push you toward alpha or a weighted variant. A 2026 arXiv review, “Counting on Consensus”, states that the right metric depends on data type, number of annotators, and whether chance correction or missing-data handling matters. The same codes can yield different values under different metrics, so the metric name belongs in every report.
When the Kappa Paradox Appears
Two documented drivers produce low kappa alongside high observed agreement. The first is prevalence: when one category dominates, p_e is high, and κ is compressed even when p_o is high. The second is rater bias: raters whose marginal distributions differ can shift p_e and change κ. A ScienceDirect abstract in the Journal of Clinical Epidemiology cites Feinstein and Cicchetti and Byrt et al. for examples in which observed agreement of 0.85 and 0.60 appear alongside unexpected kappa behaviour. I have seen only the abstract, so I cannot confirm the full framing. Zec et al. (2017) describe low kappa despite strong agreement under specific conditions.
Remedies are contested. Zec et al. and Dettori and Norvell (2020) are cited as reporting that Gwet's AC1 is more robust to the paradox. A 2025 PubMed Central paper argues that Gwet's chance-agreement formula is biased at intermediate agreement levels and proposes a maximum-likelihood kappa. Whether AC1 or alpha performs better under prevalence imbalance is not settled. A PLOS ONE paper argues kappa should be avoided as a classification performance measure and proposes prevalence-adjusted bias-adjusted kappa (PABAK) as a companion statistic.
Thresholds Are Conventions
Landis and Koch bands, such as 0.61 to 0.80 labelled “substantial”, are widely used. I did not retrieve the original paper, and the band labels here come from secondary references. Krippendorff's cutoffs of 0.800 for firm conclusions and 0.667 for tentative ones circulate through CASRAI and ResearchGate, both secondhand. I have not verified them against the 2004 text, and their page and wording are unconfirmed. The 2026 review notes that thresholds are not universally accepted and that fixed cutoffs can be too rigid for subjective tasks. Treat any single value as conditional on its design, prevalence and stakes, not as a pass or fail line.
What Agreement Cannot Establish
Agreement measures reliability, the consistency of a measurement procedure. Validity, whether labels match the truth, requires an independent reference. Kappa-type statistics pass shared bias: a team that all misreads what counts as “helpful” will agree with itself. A TianPan practitioner post gives this example. My reading, not a finding, is that low agreement more often signals ambiguous guidelines than careless annotators, and that is worth testing on your own data.
For LLM annotators and LLM-as-judge pipelines, the evidence is preprint-level. A 2026 arXiv preprint argues that LLM coders can agree with humans while measuring something else, and that for LLM coding instruments agreement is reliability, not validity. A 2024–25 preprint notes that human reference labels are themselves imperfect, so gains in LLM–human agreement may not reflect gains in truth. I did not find a settled criterion for when agreement with an LLM counts as evidence of validity.
Structured and Span Annotation Gaps
Kappa-family measures were built for flat categories. A 2024 arXiv paper (2407.11371) claims that named-entity recognition lacks an appropriate chance-corrected metric and proposes one. An ACL paper (P14-1088) reports that chance-corrected measures were rare in dependency-parsing evaluations, which used uncorrected accuracy or F1. I have not verified whether this gap persists.
What to Report and What Remains Open
Across the sources, the advice is consistent but is not a formal standard. Report observed agreement alongside a chance-corrected statistic. Name the coefficient and the number of raters. Report a prevalence-adjusted companion when one category dominates. Test whether your conclusions depend on the distinctions where raters disagree. Analyse where disagreement clusters rather than averaging it away.
Three questions remain open: whether AC1 or alpha is preferable under prevalence imbalance; what validity criterion applies to LLM coders; and what the 2024 Klie et al. study found about reporting practice. That last finding reaches me only through a 2026 review, so I cannot vouch for it. Before you publish a kappa, state the design conditions that produced it.