Temperature Scaling and ECE: Calibrate Model Confidence Step by Step

Six predictions can be correct four times out of six and still tell two very different probability stories. In the worked example below, the winning class never changes. Dividing every logit by a fitted temperature only lowers or raises the confidence attached to each winner.

That small adjustment improves mean negative log-likelihood from $0.823$ to $0.812$ . Then something awkward happens: expected calibration error, computed with four equal-width bins, rises from $12.9\%$ to $38.0\%$ .

The arithmetic is not a failure of temperature scaling. It exposes a more useful lesson: calibration is a relationship between predicted probabilities and observed outcomes, while ECE is one sample-dependent, bin-dependent estimate of that relationship. By the end, you will be able to fit the temperature, calculate ECE, reproduce both results in Python, and explain why neither number can travel unchanged into a new deployment population.

Current evaluation work makes the distinction practical. A Google DeepMind pilot published August 27 kept proprietary model weights and external test prompts hidden from each other, while MLCommons supplied a reserved AILuminate subset. Those controls protect the test material. They do not decide which population a probability should describe. NIST AI 800-3 makes the parallel point for accuracy: performance on fixed benchmark questions and performance across a broader universe of similar questions are different measurement targets.

A probability forecast has a testable promise

Suppose a classifier predicts class $A$ with confidence $0.8$ on many independent examples. A calibrated confidence means that about $80\%$ of those predictions are correct. More formally, if $\hat{Y}$ is the predicted class and $\hat{P}$ its reported confidence, perfect top-label calibration asks for

\Pr(Y=\hat{Y}\mid \hat{P}=p)=p.

This definition does not say the classifier is accurate. A model that predicts at confidence $0.6$ and succeeds $60\%$ of the time is calibrated even though it is wrong often. A model can also rank every example correctly while reporting $0.99$ for cases that succeed only $80\%$ of the time. Accuracy asks who won; calibration checks whether the attached probability kept its promise.

The event and population need names. “Correct” might mean exact class match, task completion, or another scored outcome. The examples might be customer support requests in one language, benchmark questions, or tomorrow’s production traffic. Change either side and the probability statement changes with it. For the wider evaluation pipeline behind that choice, see how evaluations shape AI products.

Temperature turns the confidence dial

A classifier often emits logits: unrestricted scores $z_1,\ldots,z_K$ for $K$ classes. Softmax converts them into probabilities:

p_k=\frac{e^{z_k}}{\sum_{j=1}^{K}e^{z_j}}.

Temperature scaling introduces one positive number $T$ :

p_k(T)=\frac{e^{z_k/T}}{\sum_{j=1}^{K}e^{z_j/T}}.

When $T>1$ , the gaps between scaled logits shrink and the probability vector becomes less concentrated. When $0<T<1$ , the gaps widen and the vector becomes sharper. $T=1$ leaves the original probabilities unchanged.

For the first example, the logits are $[3,1,0]$ . Subtracting the largest logit before exponentiating gives the same softmax without handling unnecessarily large numbers:

\operatorname{softmax}([3,1,0]) =\frac{[1,e^{-2},e^{-3}]}{1+e^{-2}+e^{-3}} \approx[0.844,0.114,0.042].

At $T=1.23$ , the vector becomes approximately $[0.779,0.153,0.068]$ . Class $A$ still wins. In fact, every positive temperature preserves the ordering of the logits, so temperature scaling cannot change the predicted class or the classification accuracy. It changes only the probability concentration.

Fit one scalar against held-out labels

Guo and colleagues’ peer-reviewed calibration paper fits $T$ by minimizing negative log-likelihood (NLL) on a held-out validation set while keeping the network parameters fixed. For $n$ labeled examples, the objective is

\mathcal{L}(T)=-\frac{1}{n}\sum_{i=1}^{n}\log p_{i,y_i}(T),

where $p_{i,y_i}(T)$ is the scaled probability assigned to the true class of example $i$ . A confident mistake receives a large penalty because the true class probability is small.

Here is the complete synthetic validation set. The labels $A$ , $B$ , and $C$ stand for three arbitrary classes; no private or high-stakes predictions are involved.

Example	Logits	True class	Predicted class	Correct?	Confidence at $T=1$	Confidence at $T=1.23$
1	$[3,1,0]$	A	A	Yes	0.844	0.779
2	$[2.5,1,0]$	A	A	Yes	0.766	0.701
3	$[3,0.5,0]$	B	A	No	0.883	0.821
4	$[1.5,2,0]$	B	B	Yes	0.574	0.537
5	$[0.5,0,1.5]$	C	C	Yes	0.629	0.575
6	$[2,2.2,0]$	A	B	No	0.518	0.496

Searching positive temperatures gives $T\approx1.22975$ . The mean NLL falls from $0.82278$ at $T=1$ to $0.81243$ at the fitted value. Accuracy remains $4/6$ , exactly as the preserved class ordering predicts.

This set is intentionally tiny enough to calculate. It is not large enough to approve a deployed calibrator, and using the same six rows both to fit $T$ and to report performance would give an optimistic estimate of generalization.

ECE compresses a reliability diagram

A reliability diagram, also called a calibration curve, groups predictions by confidence. For each bin $B_m$ , calculate

\operatorname{acc}(B_m) =\frac{1}{|B_m|}\sum_{i\in B_m}\mathbf{1}(\hat{y}_i=y_i)

and

\operatorname{conf}(B_m) =\frac{1}{|B_m|}\sum_{i\in B_m}\hat{p}_i.

The indicator $\mathbf{1}(\hat{y}_i=y_i)$ is $1$ for a correct prediction and $0$ otherwise. A bin sits on the ideal diagonal when its accuracy equals its mean confidence.

Expected calibration error takes a sample-weighted average of the absolute bin gaps:

\operatorname{ECE} =\sum_{m=1}^{M}\frac{|B_m|}{n} \left|\operatorname{acc}(B_m)-\operatorname{conf}(B_m)\right|.

The calculation below uses four fixed, equal-width confidence intervals: $[0,0.25)$ , $[0.25,0.50)$ , $[0.50,0.75)$ , and $[0.75,1]$ . Empty bins contribute nothing.

Four-bin reliability diagrams for the six-example calculation. Temperature scaling moves confidence toward the center, but the examples also cross bin edges, so the binned ECE estimate gets worse.

Exact bin statistics represented in the diagram
Temperature	Confidence interval	Examples	Mean confidence	Accuracy
1.00	[0.50, 0.75)	3	0.574	0.667
1.00	[0.75, 1.00]	3	0.831	0.667
1.23	[0.25, 0.50)	1	0.496	0.000
1.23	[0.50, 0.75)	3	0.604	1.000
1.23	[0.75, 1.00]	2	0.800	0.500

Before scaling, the three predictions in $[0.50,0.75)$ contribute

\frac{3}{6}|0.667-0.574|\approx0.047,

and the three in $[0.75,1]$ contribute about $0.082$ . Their sum is $\operatorname{ECE}\approx0.129$ , or $12.9\%$ .

After scaling, one prediction crosses below $0.50$ and the other examples form groups of three and two. The three bin contributions are approximately $0.083$ , $0.198$ , and $0.100$ , giving $\operatorname{ECE}\approx0.380$ , or $38.0\%$ .

Better likelihood, worse ECE

The fitted temperature optimized NLL, not four-bin ECE. NLL evaluates the full probability assigned to every true class and changes smoothly with $T$ . ECE throws the predictions into intervals, discards within-bin detail, and changes abruptly when a confidence crosses an edge.

Six examples amplify that instability. One mistake in a one-example bin has $0\%$ accuracy; one correct result would make the same bin $100\%$ accurate. Moving the edge or choosing a different number of bins can produce a different ECE without changing a single prediction. Guo et al. explicitly note that ECE is a binned approximation and that results can be affected by the binning scheme. scikit-learn’s calibration guide likewise pairs the curve with a histogram because a point without its sample count is easy to overread.

So the example supports three separate statements:

the selected $T$ improves NLL on these six rows;
temperature scaling leaves their predicted classes and accuracy unchanged;
four-bin ECE gets worse on the same rows.

None implies that the calibrator will improve probabilities for a fresh sample. That question needs untouched evaluation data drawn from the population the deployment decision concerns.

Reproduce every number with NumPy

The following program performs a dense search over one positive scalar. The grid is deliberately transparent; production code can use a bounded scalar optimizer and should retain an independent final evaluation set.

import numpy as np

logits = np.array([
    [3.0, 1.0, 0.0],
    [2.5, 1.0, 0.0],
    [3.0, 0.5, 0.0],
    [1.5, 2.0, 0.0],
    [0.5, 0.0, 1.5],
    [2.0, 2.2, 0.0],
])
labels = np.array([0, 0, 1, 1, 2, 0])

def probabilities(temperature):
    scaled = logits / temperature
    scaled -= scaled.max(axis=1, keepdims=True)
    exp = np.exp(scaled)
    return exp / exp.sum(axis=1, keepdims=True)

def nll(temperature):
    p = probabilities(temperature)
    return -np.log(p[np.arange(len(labels)), labels]).mean()

temperatures = np.linspace(0.25, 4.0, 15_001)
temperature = temperatures[np.argmin([nll(t) for t in temperatures])]

def ece(temperature, edges=np.linspace(0.0, 1.0, 5)):
    p = probabilities(temperature)
    predictions = p.argmax(axis=1)
    confidence = p.max(axis=1)
    correct = predictions == labels
    total = 0.0
    for lower, upper in zip(edges[:-1], edges[1:]):
        in_bin = (confidence >= lower) & (confidence < upper)
        if upper == 1.0:
            in_bin = (confidence >= lower) & (confidence <= upper)
        if in_bin.any():
            total += in_bin.mean() * abs(
                correct[in_bin].mean() - confidence[in_bin].mean()
            )
    return total

print(f"T={temperature:.5f}")
print(f"NLL: {nll(1.0):.5f} -> {nll(temperature):.5f}")
print(f"ECE: {ece(1.0):.3f} -> {ece(temperature):.3f}")

The expected output is approximately:

T=1.22975
NLL: 0.82278 -> 0.81243
ECE: 0.129 -> 0.380

A calibrator belongs to one measurement contract

A defensible calibration record should name the model checkpoint, unscaled logits, outcome definition, population, data windows, split method, fitted $T$ , optimization objective, reliability-bin edges, bin counts, and at least one proper scoring rule such as NLL. Fit the calibrator on examples that were not used to train the classifier, then assess it on untouched data.

Distribution shift is the decisive boundary case. A temperature fitted on clean benchmark questions may be wrong for short prompts, another language, a new class mix, or inputs collected after the product changes. Temperature scaling cannot repair a bad class ranking, detect an unseen domain, or prove a system safe. It only rescales the logits under the validation assumptions used to fit one number.

That is why a deployment review should keep the reliability curve and sample counts beside ECE rather than treating the scalar as a certificate. If a new cohort moves those points away from the diagonal, the next task is not to defend the old ECE. It is to identify which population changed, collect enough labels to measure it, and decide whether the calibration contract still describes the system in use.

A probability forecast has a testable promise

Temperature turns the confidence dial

Fit one scalar against held-out labels

ECE compresses a reliability diagram

Better likelihood, worse ECE

Reproduce every number with NumPy

A calibrator belongs to one measurement contract

Sources

Follow new Math for AI articles

Keep reading

ROC-AUC at 0.97, Precision at 9%: The Base-Rate Math