Skip to content
    Flow-alerta.cab

    Expected Calibration Error

    AI fundamentals & modelsECE
    Editorially reviewed on: 1 source
    Your reading view

    The short answer · 11 words

    In short

    ECE summarizes how strongly predicted confidences differ from observed hit frequencies.

    Flow-alerta.cab (2026): Expected Calibration Error. https://flow-alerta.cab/en/glossar/expected-calibration-error. Editorially reviewed on: 10 October 2026.

    How much detail would you like?

    Practical

    Calibration supports thresholds, risk communication and manual escalation.

    Hypothetical example: A classifier is recalibrated on a separate calibration data set and then evaluated on independent test data.

    SMEs should use ECE alongside discrimination metrics, reliability diagrams and business consequences.

    At a glance

    Core
    Predictions are usually divided into bins according to their confidence. For each bin, the absolute difference between mean confidence and observed accuracy is calculated and weighted by the number of his cases. Result and statement depend on binning and variant.
    Use
    Calibration supports thresholds, risk communication and manual escalation.
    Example
    Hypothetical example: A classifier is recalibrated on a separate calibration data set and then evaluated on independent test data.
    Limits
    Interpret low ECE as a good classification; choose bins at will; ignore subgroups and distribution shift.

    In context

    Brier Score

    Proper scoring rule for evaluating probabilistic predictions based on the quadratic deviation between predicted probabilities and actual results.

    Decision Threshold

    The decision threshold translates a model score into a class or action.

    Questions about this term

    Can an inaccurate model be well calibrated?

    Yes.

    Is ECE clearly defined?

    No, variants are different.

    All details and background

    Predictions are usually divided into bins according to their confidence. For each bin, the absolute difference between mean confidence and observed accuracy is calculated and weighted by the number of his cases. Result and statement depend on binning and variant.

    ELI5 – Explained simply

    If a model says 80 percent, it should be about eight out of ten times right in the long run.

    Why it matters in practice

    Calibration supports thresholds, risk communication and manual escalation.

    Example

    Hypothetical example: A classifier is recalibrated on a separate calibration data set and then evaluated on independent test data.

    Pitfalls and limits

    • Interpret low ECE as a good classification; choose bins at will; ignore subgroups and distribution shift.

    Origin & history

    Calibration measures are based on probabilistic prediction; ECE has been disseminated in modern deep learning literature.

    Use in mid-sized companies

    • SMEs should use ECE alongside discrimination metrics, reliability diagrams and business consequences.

    Related terms

    Matching Flow-alerta service

    AI opportunity analysis & roadmapAI consulting: test confidence levels, calibration and appropriate decision thresholds of a model.

    Matching tool / use case

    View use casesUse Case: Calibration of risk scores before threshold decisions

    Sources

    Quellenprüfung: 10.10.2026. Definition und dokumentierte Einordnung abgeglichen; Beispiele beschreiben mögliche Anwendungen.

    Editorially reviewed on:

    Next step

    Put knowledge into practice

    Discuss your projectView use cases