A classifier is well calibrated when its predicted probabilities coincide, on average, with the fraction of objects that truly belong to the predicted class. The miniJPAS survey (arXiv:2303.12684, Section 3.5.3) gives this as the standard definition. Calibration is separate from accuracy. A model can label most cases correctly while its confidence values are systematically too high, and a model can be well calibrated without being highly accurate.
What calibration means for a classifier
Calibration concerns the meaning of the probability output, not whether the top label is right. A set of predictions made with confidence 0.8 should be correct about 80% of the time, measured over many predictions. That is the property every metric below tries to estimate.
Reliability diagrams: construction and reading
A reliability diagram groups predictions into bins by predicted confidence. For each bin it plots observed accuracy against mean confidence (arXiv:1904.01685). A perfectly calibrated model traces the diagonal. Points below the diagonal indicate over-confidence, meaning the model is more confident than it is accurate. Points above the diagonal indicate under-confidence. As a practice recommendation of my own, show the sample count for each bin next to the plot. A sparse bin produces a noisy point that can look like a large deviation.
Expected calibration error: computation step by step
Guo et al. (ICML 2017; arXiv:1706.04599) defined the binned estimate now common in practice. The steps are:
- Take n validation predictions, each with a predicted label and a confidence.
- Partition the confidence range from 0 to 1 into M interval bins B_m.
- For each bin, compute accuracy acc(B_m), the fraction of predictions whose label is correct, and conf(B_m), the mean confidence.
- Compute ECE as the sum over m of (|B_m| / n) multiplied by the absolute value of acc(B_m) minus conf(B_m).
ECE is therefore a weighted average of the bin gaps. Empty bins contribute nothing. The value is an estimate computed on a finite sample, not a true expectation. The 2023 Springer survey (doi 10.1007/s10994-023-06336-7) argues that the binary version should be called an 'estimated' calibration error, and the 2025 review (arXiv:2504.18278) notes that many authors use 'empirical' or 'estimated' for the same reason.
Choosing the bin count
M is a free choice, not a standard. Ten bins appear in one paper (arXiv:1401.3390) and twenty equal-width bins in another (arXiv:1912.03263). A fifteen-bin setting is also attributed to Guo et al. in a secondary source that I did not check against the original PDF. Report M with every ECE value, and check whether conclusions change when M changes.
Multiclass: which confidence is used
For multiclass classifiers, the most common convention is top-label: the confidence is the maximum softmax probability, and accuracy asks whether the top class is correct. A secondary summary (Emergent Mind's ECE topic page) describes this convention. Guo et al. use the predicted-class probability. Because the survey defines several variants (below), state which one you report.
Maximum calibration error
Maximum calibration error (MCE) replaces the weighted average with the largest absolute bin gap: MCE is the maximum over m of the absolute value of acc(B_m) minus conf(B_m). Guo et al. motivate it for high-risk applications where worst-case deviation matters more than average deviation (PMLR version, ICML 2017). In my reading, MCE is the better choice when a small confidence region is dangerous. It is also noisier, because one sparse bin with a large gap sets the value. Report bin counts alongside it.
Multiclass calibration and its variants
Top-label calibration checks only the predicted class. Classwise calibration checks each class probability separately. The Springer survey (Section 4) gives several definitions. I did not find a single authoritative convention, so a top-label ECE does not show that the other class probabilities are calibrated.
Proper scoring rules and their calibration component
The Brier score and log-loss are proper scoring rules, meaning they are minimised in expectation when the reported probabilities are the true ones. The Springer survey (Section 3.1) decomposes such scores into a calibration loss plus a refinement loss, which reflects discrimination. This decomposition supports reporting a calibration component alongside the overall score. The survey's numerical example is illustrative and is not a benchmark.
Known limitations of ECE and proposed alternatives
- Binning sensitivity. The value changes with M and with the bin edges.
- Trivially perfect ECE. The 2025 review (arXiv:2504.18278, Section 4.2) reports that a perfect ECE can be obtained by estimating probabilities from the label distribution, so a low ECE alone does not show a useful model. The review attributes this to another paper, which I did not retrieve.
- Local miscalibration. ECE can miss miscalibration confined to a region of confidence or to a subgroup. This is reported in a secondary summary.
- Measurement across bins only. Measuring Calibration in Deep Learning (arXiv:1904.01685) argues that ECE has numerous flaws and recommends alternatives. I did not retrieve its full list.
Alternatives vary in evidence strength. Kernel density estimators are reported to yield lower relative errors, especially for confidence calibration. That claim rests on a secondary summary citing Posocco et al. (2021), which I have not checked against the original. V-ECE, a recent arXiv preprint (arXiv:2602.24230), fits recalibration functions with cross-validation and reports strong accuracy on semi-synthetic benchmarks. It is a single preprint and, as far as I could determine, not peer-reviewed.
Open questions and reporting practice
Several points remain unsettled in the material I reviewed. The number of validation points a stable ECE needs is not established, and I found no practitioner guide to confidence intervals for ECE. Interaction with class imbalance is thinly covered. No authoritative standard exists for which bin count to state or which companion metrics to publish.
Guo et al. found that depth, width, weight decay and Batch Normalization all influence calibration. Modern networks were poorly calibrated, while LeNet-style networks were close to calibrated. Temperature scaling, a single-parameter variant of Platt scaling, was surprisingly effective on most datasets tested (arXiv:1706.04599; PMLR version).
A practical reporting checklist, which is my recommendation and not a sourced standard: state M and per-bin counts, report MCE or a proper scoring rule alongside ECE, and report accuracy too. Fit any recalibration on data separate from the evaluation data. This last step is my synthesis; no source I reviewed states it explicitly, and it is the step most often skipped.
For sample-size questions, see statistical power when comparing machine learning models. For summary metrics that hide structure, see what inter-annotator agreement measures and misses.