L1, L2, and L-infinity See Size Differently

A model can be sensitive to a change that looks tiny in every input coordinate. To describe that change, we first have to choose how to measure its size. The L1, L2, and L-infinity norms give different answers to that question—and their unit balls show exactly why.

By the end, you will be able to compare those norms, read their two-dimensional shapes, and tell a penalty on model parameters apart from a budget on input changes.

Learning outcome: Explain how different norms change geometry and sensitivity.

Lesson 39 introduced norms as rules for measuring vectors. Section 3.1 of the official Mathematics for Machine Learning book develops common norms and their geometry. The examples, comparison, interactive figure, and AI applications below are original.

One vector, three measurements

For a real vector $x=(x_1,\ldots,x_n)$ , the L1 norm adds the absolute coordinate values, the L2 norm takes the square root of their squared sum, and the L-infinity norm keeps the largest absolute coordinate:

Take $x=(3,-4)$ . Its L1 norm is $|3|+|-4|=3+4=7$ . Its L2 norm is $\sqrt{3^2+(-4)^2}=\sqrt{9+16}=5$ . Its L-infinity norm is $\max(3,4)=4$ . Nothing about the vector changed; each rule emphasizes a different aspect of its coordinates. L1 accumulates all coordinate magnitudes, L2 measures Euclidean length, and L-infinity reports the single largest change.

NumPy can compute each vector norm by setting ord explicitly:

import numpy as np

x = np.array([3.0, -4.0])
np.linalg.norm(x, ord=1)      # 7.0
np.linalg.norm(x, ord=2)      # 5.0
np.linalg.norm(x, ord=np.inf) # 4.0

Here x is one-dimensional, so ord=2 requests its Euclidean norm. With a two-dimensional array, numpy.linalg.norm can instead compute a matrix norm; specify both the norm order and the vector axis when working with batches of vectors. NumPy documents these vector and matrix cases separately.

Unit balls make the choice visible

A unit ball for a norm is the set of vectors whose norm is at most one. In two dimensions, the L1 unit ball is a diamond, the L2 unit ball is a circle, and the L-infinity unit ball is an axis-aligned square. For example, $(0.6,0.8)$ lies exactly on the L2 boundary because $\sqrt{0.6^2+0.8^2}=1$ . It sits outside the L1 ball because $0.6+0.8=1.4$ , and inside the L-infinity ball because its largest coordinate magnitude is $0.8$ .

Move either coordinate below. The same point is tested against all three boundaries, while the table reports each norm and whether the point is inside its unit ball.

One vector inside three differently shaped unit balls

Choose the vector coordinates

Use the arrow keys to change a coordinate by 0.1. Each norm is measured on the same vector.

First coordinate, x₁0.6Second coordinate, x₂0.8

For x = (0.6, 0.8): L1 is 1.4 (outside); L2 is 1 (on the boundary); L-infinity is 0.8 (inside).

Unit-ball boundaries

L1: diamond boundary
L2: circle boundary
L-infinity: square boundary

Norm values and unit-ball membership for the selected vector
Norm	Value	Position
L1	1.4	outside
L2	1	on the boundary
L-infinity	0.8	inside

The status table remains available without interpreting the plotted colors or shapes.

The shape determines which directions fit under a fixed budget. At $(0.7,0.7)$ , for instance, L-infinity is $0.7$ and L2 is about $0.99$ , so the point is inside both unit balls; L1 is $1.4$ , so it falls outside the diamond. A square budget limits each coordinate separately, while a round budget limits their combined Euclidean length.

The norms still constrain one another. For every $x\in\mathbb{R}^n$ ,

\lVert x\rVert_{\infty}\leq\lVert x\rVert_2\leq\lVert x\rVert_1 \leq\sqrt{n}\,\lVert x\rVert_2\leq n\,\lVert x\rVert_{\infty}.

The largest coordinate cannot exceed the square root of the sum of all squared coordinates, which cannot exceed the sum of the absolute coordinates. The Cauchy–Schwarz inequality gives the next bound, $\lVert x\rVert_1\leq\sqrt{n}\lVert x\rVert_2$ ; bounding every squared coordinate by the largest one gives $\lVert x\rVert_2\leq\sqrt{n}\lVert x\rVert_{\infty}$ . Thus the measures differ by factors that can grow with the dimension, and can rank two vectors differently. For $u=(3,0)$ and $v=(2,2)$ , L1 says $v$ is larger ( $4>3$ ), while L2 says $u$ is larger ( $3>\sqrt{8}$ ).

Parameter penalties and input budgets answer different questions

During model fitting, regularization adds a term to the objective to discourage certain parameter values. A training objective might include $\lambda_1\lVert w\rVert_1$ or $\lambda_2\lVert w\rVert_2^2$ , where $w$ is a vector of learned parameters and each $\lambda$ sets the penalty strength. L1 penalties can favor sparse coefficients; squared L2 penalties tend to shrink many coefficients without generally making them zero. The feature scales and penalty strengths affect the result. The squared L2 expression is a squared norm, not itself a norm.

An input perturbation budget applies a norm to a different vector. If $\delta$ is added to input $x$ , the constraint $\lVert\delta\rVert_{\infty}\leq\varepsilon$ means that every coordinate changes by at most $\varepsilon$ . If image values have been scaled to $[0,1]$ , then $\varepsilon$ is measured on that scale; clipping a modified image back into the valid input range is a separate constraint.

Goodfellow, Shlens, and Szegedy’s fast gradient sign method uses this L-infinity geometry. For an untargeted attack, let $J(\theta,x,y)$ be the model’s loss at parameters $\theta$ , input $x$ , and label $y$ . A single step is

\delta=\varepsilon\,\operatorname{sign}\!\left(\nabla_xJ(\theta,x,y)\right).

The gradient estimates how a small input change affects the loss. Choosing each coordinate’s sign in that direction maximizes the first-order increase while keeping every coordinate within $\varepsilon$ . This is a local one-step construction: it does not guarantee a misclassification, and its success depends on the model and input. Nor does a small L-infinity value guarantee that a change is imperceptible or realistic. The bound states a coordinate-wise budget; people must decide whether that budget matches the threat they care about.

When a model’s sensitivity is being evaluated, record what vector may change, which norm constrains it, and the units of its budget. Then a statement such as “robust within $\varepsilon$ ” has a precise geometric meaning instead of hiding its assumptions in the word small.