Vector Norms Measure Size, Not Meaning

Two embedding vectors can point in the same direction and still have different magnitudes. For example, $(3,4)$ and $(6,8)$ are scaled copies: the second is twice as long. A norm turns a vector into a nonnegative number that summarizes its size according to a specified rule. That number can help compare scales or describe a parameter penalty, but it does not rank how meaningful an embedding is.

By the end of this lesson, you will be able to calculate the $L_1$ , $L_2$ , and $L_\infty$ vector norms, check the properties that make a function a norm, and use NumPy to compute vector sizes without confusing a batch of vectors for one matrix.

Learning outcome: Compute common vector norms and check the norm properties.

This lesson builds on Lesson 014: Vectors as AI Data, where coordinates were features in a fixed order. Section 3.1 of Mathematics for Machine Learning introduces norms as ways to measure vector length. The examples, code, visual, and assessments here are original.

A norm is a size rule with three promises

Let $x$ and $y$ be vectors in $\mathbb{R}^n$ , and let $c$ be any real number. A function $\lVert\cdot\rVert$ is a vector norm if it satisfies all three rules:

The first rule says a vector’s size cannot be negative, and only the zero vector has size zero. The second says scaling a vector by a negative number still multiplies its size by the scalar’s absolute value. The third says combining two vectors cannot produce a size larger than the sum of their separate sizes.

Three common rules applied to one vector

For $x=(x_1,\ldots,x_n)$ , the common norms are

\lVert x\rVert_1=\sum_{i=1}^n |x_i|,\qquad \lVert x\rVert_2=\sqrt{\sum_{i=1}^n x_i^2},\qquad \lVert x\rVert_\infty=\max_i |x_i|.

The subscript selects the rule. $L_1$ adds absolute coordinate sizes, $L_2$ uses the familiar Euclidean length, and $L_\infty$ keeps the largest absolute coordinate. These are three different summaries of the same vector; the coordinates themselves do not change.

Take the vector $x=(3,-4)$ . Its negative second coordinate matters for some operations, but each of these size rules uses absolute values or squares:

The Euclidean unit ball is the set of two-coordinate vectors whose $L_2$ norm is at most one. The point $(0.6,0.8)$ lies on its boundary because $\sqrt{0.6^2+0.8^2}=1$ .

The Euclidean unit ball contains vectors of length at most oneEvery point inside or on the circle has L₂ norm no greater than 1. The marked vector (0.6, 0.8) reaches the boundary.

L₂ unit ball: ‖x‖₂ ≤ 1

The shaded region encodes a rule, not just a shape: it contains the origin and every vector whose $L_2$ length is no greater than one. The boundary is where the length equals one. In higher dimensions the same set is called a ball too, even though it is no longer something we can draw as a disk.

Check the three promises

Positive definiteness follows directly for each formula. A sum of absolute values or squares can be zero only when every coordinate is zero. A maximum of absolute values is zero only when every coordinate is zero. Each formula is nonnegative, and each returns zero for the zero vector.

For absolute homogeneity, take $x=(3,-4)$ and multiply it by $c=-2$ :

-2x=(-6,8),\qquad \lVert -2x\rVert_1=14=2\cdot7,\quad \lVert -2x\rVert_2=10=2\cdot5,\quad \lVert -2x\rVert_\infty=8=2\cdot4.

The absolute value on $c$ is essential. A negative scalar flips the vector’s direction, but length does not become negative.

The triangle inequality can also be checked from the formulas. For $L_1$ , the coordinate rule $|x_i+y_i|\le |x_i|+|y_i|$ holds for each coordinate, so summing those inequalities gives $\lVert x+y\rVert_1\le\lVert x\rVert_1+\lVert y\rVert_1$ . For $L_2$ , it is the familiar geometric fact that one side of a triangle is no longer than the other two sides combined. For $L_\infty$ ,

\lVert x+y\rVert_\infty =\max_i|x_i+y_i| \le\max_i(|x_i|+|y_i|) \le\max_i|x_i|+\max_i|y_i|.

Each norm therefore passes all three checks. A single numerical example can test whether you understand a rule, while the reasoning above explains why the rules hold for every vector.

Compute vector sizes with NumPy

NumPy exposes these calculations through numpy.linalg.norm. For a one-dimensional array, specify ord to choose a vector norm:

import numpy as np

x = np.array([3.0, -4.0])
print(np.linalg.norm(x, ord=1))
print(np.linalg.norm(x, ord=2))
print(np.linalg.norm(x, ord=np.inf))

The output is 7.0, 5.0, and 4.0. Omitting ord for a one-dimensional array defaults to the $L_2$ norm.

An embedding table is often a two-dimensional array, with one embedding per row. To get one $L_2$ norm per embedding, specify the row axis:

embeddings = np.array([[3.0, 4.0], [6.0, 8.0]])
row_norms = np.linalg.norm(embeddings, ord=2, axis=1)
print(row_norms)

The output is [5. 10.]. The second vector has twice the first one’s norm, even though both point in the same direction. With a two-dimensional input and no axis, NumPy interprets the input as a matrix; a matrix norm is not the same operation as calculating a vector norm for each row. The documentation describes axis as the choice of dimension along which vector norms are computed.

A vector norm says how large a representation is under a chosen coordinate rule. It does not say the larger embedding is more relevant, accurate, or semantically important. Those conclusions require a model-specific task and evaluation; the scalar magnitude alone cannot supply them.

An optimization objective can add a term that penalizes large parameter vectors $w$ . For example, an $L_1$ penalty adds $\lambda\lVert w\rVert_1$ , while a squared $L_2$ penalty adds $\lambda\lVert w\rVert_2^2$ , with $\lambda\ge0$ controlling the penalty’s weight. For $w=(3,-4)$ , $\lVert w\rVert_1=7$ while $\lVert w\rVert_2^2=25$ .

The square in the second expression is important: $\lVert w\rVert_2^2$ is not itself a norm. Norm homogeneity scales it by $c^2$ , not by $|c|$ : $\lVert cw\rVert_2^2=c^2\lVert w\rVert_2^2$ . The two penalty formulas also have different numerical scales, so choosing the same $\lambda$ does not make their effects comparable. The vector norm explains what a penalty measures; the objective and its coefficient determine how that measurement affects training.

One software detail deserves care. NumPy’s ord parameter supports values below 1 for numerical purposes, but its documentation explicitly says those results are not mathematical norms. For example, if $f(x)=(\sum_i|x_i|^{1/2})^2$ , then $f((1,0))=1$ and $f((0,1))=1$ , while $f((1,1))=4$ . Since $4>1+1$ , the triangle inequality fails. Use ord=0.5 when you specifically need that numerical expression, not when you need a vector norm with the three guarantees above.

With the promises and the implementation distinction clear, the next step is to compare the geometries different valid norms create—and how those choices change notions of distance and constraint.