Distance depends on the chosen geometry

When a support-ticket router, recommender, or retrieval system looks for the “closest” stored example, it is making a geometric choice. A point that is closest when every coordinate contributes separately can lose to a different point when the coordinates combine by squared length. The algorithm did not become inconsistent; the meaning of “close” changed.

By the end of this lesson, you will be able to derive a distance from a norm, calculate a small nearest-neighbor ranking under several metrics, and identify the preprocessing and tie-handling decisions that make that ranking meaningful.

Learning outcome: Derive distance from a norm and compare nearest neighbors.

This lesson builds on Lesson 040: L1, L2, and L-infinity See Size Differently, which introduced several ways to measure the size of one vector. Section 3.3 of the official Mathematics for Machine Learning book supplies the mathematical reference. The dataset, calculations, code, visual, and assessments here are original.

The closest point starts with a displacement

Let $x=(x_1,\ldots,x_n)$ and $y=(y_1,\ldots,y_n)$ be two points in $\mathbb{R}^n$ . Each coordinate is a feature position; $n$ is the number of features. To ask how far $x$ is from $y$ , first subtract coordinate by coordinate:

x-y=(x_1-y_1,\ldots,x_n-y_n).

This difference is a displacement vector. It says how much one point must move in each coordinate to land on the other. The order changes the sign of the displacement, but a norm gives the same size either way because $\lVert v\rVert=\lVert -v\rVert$ .

The names describe different movement rules. L1, often called Manhattan distance, adds the amount moved in every coordinate. L2 measures the straight line length of the displacement. L-infinity only asks for the largest single coordinate movement. A model that uses one of these measures is declaring which kinds of movement count as expensive.

The Section 3.3 reference also connects an inner product to a norm through $\lVert v\rVert=\sqrt{\langle v,v\rangle}$ . With the usual dot product, this is the Euclidean norm, so $d_2(x,y)=\sqrt{\langle x-y,x-y\rangle}$ . A useful subtlety is that the distance construction works for any norm: L1 is a valid norm and induces a valid metric even though it is not induced by an inner product in the usual real-vector-space sense.

Why subtraction must happen first

The distance compares two locations through one displacement. It is not the difference between their separate sizes. For example, if $x=(1,1)$ and $y=(1,-1)$ , both points have L2 norm $\sqrt{2}$ , but

x-y=(0,2),\qquad d_2(x,y)=\sqrt{0^2+2^2}=2.

Computing $\lVert x\rVert_2-\lVert y\rVert_2=0$ would erase the fact that the points sit two units apart vertically. The safe mental recipe is subtract, then apply the chosen norm.

The same points can have different nearest neighbors

Consider a query point $q=(0,0)$ and three candidate records in a two-feature representation. The coordinates are deliberately small and unitless: they stand for values after some chosen feature representation, not for a universal notion of similarity.

Point	Description	Coordinates
$A$	axis-heavy case	$(6,0)$
$B$	balanced case	$(4,4)$
$C$	mixed case	$(5,2)$

Calculate every distance before ranking

Because $q=(0,0)$ , each displacement is the candidate itself. For point $A$ ,

\begin{aligned} A-q&=(6,0),\\ d_1(q,A)&=|6|+|0|=6,\\ d_2(q,A)&=\sqrt{6^2+0^2}=\sqrt{36}=6,\\ d_\infty(q,A)&=\max(|6|,|0|)=6. \end{aligned}

For point $B$ ,

\begin{aligned} B-q&=(4,4),\\ d_1(q,B)&=|4|+|4|=8,\\ d_2(q,B)&=\sqrt{4^2+4^2}=\sqrt{32}\approx5.657,\\ d_\infty(q,B)&=\max(4,4)=4. \end{aligned}

For point $C$ ,

\begin{aligned} C-q&=(5,2),\\ d_1(q,C)&=|5|+|2|=7,\\ d_2(q,C)&=\sqrt{5^2+2^2}=\sqrt{29}\approx5.385,\\ d_\infty(q,C)&=\max(5,2)=5. \end{aligned}

Now sort each column from the smallest distance to the largest:

\begin{aligned} \text{L1:}&\quad A\;(6),\ C\;(7),\ B\;(8),\\ \text{L2:}&\quad C\;(5.385),\ B\;(5.657),\ A\;(6),\\ \text{L-infinity:}&\quad B\;(4),\ C\;(5),\ A\;(6). \end{aligned}

The ranking changes because the norms reward different shapes of displacement. L1 favors the axis-heavy point $A$ here because it has the smallest total coordinate movement. L2 picks $C$ : its two coordinates combine to a shorter straight-line segment than $B$ ‘s diagonal and $A$ ‘s long horizontal segment. L-infinity picks $B$ because no individual coordinate moves by more than $4$ .

One query, three neighbor maps

The table is the complete semantic fallback: smaller distances rank closer to the query q = (0, 0).
Candidate	Coordinates	L1sum of absolute moves	L2straight-line length	L-infinitylargest coordinate move
A axis-heavy	(6, 0)	6.000rank 1	6.000rank 3	6.000rank 3
B balanced	(4, 4)	8.000rank 3	5.657rank 2	4.000rank 1
C mixed	(5, 2)	7.000rank 2	5.385rank 1	5.000rank 2
Two nearest candidates		A → C	C → B	B → C

Numbers inside the circles give the rank for that metric. The dashed segment points to the nearest candidate; the first two ranks are outlined so the map does not rely on color alone.

For L1, the order is A then C then B. For L2, it is C then B then A. For L-infinity, it is B then C then A. The query is at the origin in every panel.

The geometry changes the neighbor map even though the query and candidate coordinates stay fixed.

The table and rank labels make the conclusion available without reading the SVG. The visual adds one spatial cue: the dashed segment changes endpoint when the metric changes, while the coordinates themselves remain fixed.

A transparent NumPy search

The NumPy numpy.linalg.norm documentation exposes the two controls needed here: ord chooses the norm order and axis chooses which coordinate axis to reduce. A batch of three candidate rows is therefore handled by subtracting the query from every row and reducing each row separately.

The following code mirrors the derivation rather than hiding it in a nearest neighbor object. It was executed with NumPy; the printed ordering is the same as the worked example.

import numpy as np

query = np.array([0.0, 0.0])
points = np.array([[6.0, 0.0], [4.0, 4.0], [5.0, 2.0]])
labels = np.array(["A", "B", "C"])

difference = points - query
for name, order in [("L1", 1), ("L2", 2), ("L-infinity", np.inf)]:
    distances = np.linalg.norm(difference, ord=order, axis=1)
    ranked = np.argsort(distances, kind="stable")
    print(
        f"{name}: "
        + ", ".join(f"{labels[i]}={distances[i]:.3f}" for i in ranked)
    )

L1: A=6.000, C=7.000, B=8.000
L2: C=5.385, B=5.657, A=6.000
L-infinity: B=4.000, C=5.000, A=6.000

There are three separate operations worth noticing:

points - query performs the subtraction before any norm is applied.
axis=1 computes one norm per row, so each candidate gets one distance.
argsort turns distances into a ranking; the norm itself does not decide how to handle equal values.

For a larger data set, scikit-learn’s NearestNeighbors class packages the same idea. Its metric="minkowski" setting uses the parameter p, with $p=1$ corresponding to L1 and $p=2$ to L2; its kneighbors method returns both distances and indices. The estimator is useful when the search should be integrated into a model pipeline, but the NumPy version is the better debugging tool because every intermediate array is visible.

For this exact example, the equivalent search configuration is:

from sklearn.neighbors import NearestNeighbors

search = NearestNeighbors(n_neighbors=2, metric="minkowski", p=2)
search.fit(points)
distances, indices = search.kneighbors(query.reshape(1, -1))
nearest_labels = labels[indices[0]]

Changing p to 1 or np.inf changes the geometry while keeping the data and the value of $k=2$ fixed. The API does not make the metric choice disappear; it makes that choice explicit in a parameter.

Boundaries that change the answer

Distance is precise only after the feature space, metric, and search policy are specified. These are the cases most likely to make a small nearest-neighbor demo misleading.

Ties are information, not noise

Let $q=(0,0)$ , $u=(1,1)$ , and $v=(2,0)$ . Both candidates have

d_1(q,u)=|1|+|1|=2, \qquad d_1(q,v)=|2|+|0|=2.

If $k=1$ , the mathematics supplies a tie, not a unique winner. A production system should state a deterministic secondary key, return all tied candidates, or use a policy appropriate to its application. The stable sort in the NumPy code makes repeated runs reproducible, but it does not prove that the first tied row is more similar.

Identical points have distance zero

If $x=y$ , then $x-y=(0,\ldots,0)$ and every norm gives $d(x,y)=0$ . That is a valid exact match, not an error. In a training-set lookup, however, querying a row against the same fitted collection can return the row itself as the nearest neighbor. Decide whether self-matches are useful before interpreting a result; an external query that happens to duplicate a stored row should still receive distance zero.

Feature scaling is part of the geometry

Suppose one coordinate is annual income in dollars and another is a count from zero to ten. The dollar coordinate can dominate an L1 or L2 distance simply because its numeric unit is larger. Standardizing, rescaling, or assigning domain weights changes the geometry; it is not a cosmetic prelude. Fit any data-dependent scaling rule on the training data and apply the same rule to queries, or the neighbor map can leak information from the evaluation set.

Distance and cosine similarity answer different questions

Distance compares the displacement between two points and therefore includes their positions and magnitudes. Cosine similarity compares direction after normalizing lengths. For $x=(1,1)$ and $y=(10,10)$ ,

\cos(x,y)=\frac{x^\mathsf{T}y}{\lVert x\rVert_2\lVert y\rVert_2}=1,

because the vectors point in exactly the same direction, while

d_2(x,y)=\lVert x-y\rVert_2 =\sqrt{(-9)^2+(-9)^2}=9\sqrt{2}\approx12.728.

An embedding-retrieval system may prefer cosine when direction is the intended notion of match. A scale-sensitive application may prefer a norm-induced distance. Neither choice is universally correct, and cosine similarity needs a separate policy for a zero vector because its direction is undefined.

High dimension calls for measurement, not a slogan

In some high-dimensional data distributions, nearest and farthest distances can become closer together, making the ranking less discriminative. Beyer, Goldstein, Ramakrishnan, and Shaft describe this distance-concentration result under specific conditions in their nearest-neighbor paper. That result is a warning about data and workload assumptions, not a law that every embedding space becomes useless. Inspect neighbor-distance gaps, validate the metric on held-out queries, and consider whether the representation contains many irrelevant coordinates before blaming the search algorithm.

The next lesson isolates the directional comparison that distance deliberately mixes with length: Lesson 044, Angles and Cosine Similarity. The Math for AI course page will show that lesson when its translation set is live.