Angles and Cosine Similarity

Two documents can be represented by vectors $x=(1,2)$ and $y=(10,20)$ . Their raw dot product is $50$ , so a dot-product search sees the second vector as much larger. Their directions are identical, though: after dividing by both vector lengths, their cosine similarity is $1$ . The scoring rule changes what “similar” means.

By the end of this lesson, you will be able to compute an angle and cosine similarity from coordinates, explain why positive rescaling leaves cosine unchanged, and handle a zero vector without assigning it a direction that it does not have.

Learning outcome: Compute an angle and cosine similarity and handle zero-vector limits.

This lesson combines two earlier ideas: Lesson 039: Norms measure vector size gives us Euclidean lengths, and Lesson 041: Inner products measure alignment gives us dot products. Lesson 043: Distance depends on the chosen geometry then showed why a distance keeps magnitude in the comparison. Section 3.4 of the official Mathematics for Machine Learning book develops the angle relationship. The examples, code, visual, and assessments here are original.

An angle separates direction from length

For two nonzero vectors $x,y\in\mathbb{R}^n$ , write $\theta$ for the angle between them. The usual dot product and Euclidean norm are

The first expression combines matching coordinates into one scalar. The second measures each vector’s Euclidean length. The third divides the alignment score by the two lengths, leaving a directional comparison. The Cauchy–Schwarz inequality guarantees that the final ratio lies between $-1$ and $1$ for nonzero vectors, so the angle is recovered unambiguously in the interval $[0,\pi]$ :

\theta = \arccos\!\left(\frac{x^\mathsf{T}y}{\lVert x\rVert_2\,\lVert y\rVert_2}\right).

This normalized value is cosine similarity. A value near $1$ means the vectors point in nearly the same direction; $0$ means a right angle under the usual dot-product geometry; and $-1$ means opposite directions. Those interpretations assume both vectors are nonzero and that we are using the standard Euclidean inner product.

A worked angle from coordinates

Take $x=(3,1)$ and $y=(1,2)$ . These are two-dimensional vectors, so each has two coordinates and no physical unit is implied. We can calculate every part before applying an inverse cosine.

The intermediate values matter. A dot product of $5$ alone does not tell us the angle, because another pair of vectors could have the same dot product but different lengths. Dividing by both norms makes the result comparable across vector magnitudes.

Positive scaling leaves cosine unchanged

Suppose a nonzero vector is stretched by a positive number $c$ . Its direction does not change, and the algebra shows why the cosine score stays fixed:

\cos(cx,y)=\frac{(cx)^\mathsf{T}y}{\lVert cx\rVert_2\lVert y\rVert_2} =\frac{c\,x^\mathsf{T}y}{c\lVert x\rVert_2\lVert y\rVert_2} =\cos(x,y),\qquad c>0.

The same cancellation works when $y$ is positively scaled. This is the useful feature for directional embedding comparisons: a document vector with the same orientation but a larger norm receives the same cosine score. It is also a modeling choice. If magnitude carries signal about confidence, frequency, or another property of the representation, removing it can discard information.

Explore the angle, score, and lengths together

The explorer starts with two nonzero vectors. Move any coordinate with a mouse, touch input, or arrow keys. The plot shows the same vectors from the origin, while the table repeats the dot product, both norms, cosine similarity, and recovered angle in text. Set either vector to $(0,0)$ to see the zero-vector boundary.

Cosine similarity changes when vector direction changes

Choose the two vector coordinates

Each range control changes one coordinate by 0.5. The table is the complete text alternative to the plotted geometry.

Vector x

First coordinate1Second coordinate0.5

Vector y

First coordinate0.5Second coordinate1.5

x = (1, 0.5); y = (0.5, 1.5); dot product = 1.25; norm of x = 1.118; norm of y = 1.581; cosine similarity = 0.707; angle = 0.785 radians (45 degrees). Interpretation: the score reflects partial directional alignment.

Text values for the selected vectors
Quantity	Value
Dot product x · y	1.25
Norm of x	1.118
Norm of y	1.581
Cosine similarity	0.707
Angle	0.785 radians / 45 degrees

The vector lines use distinct dash patterns and labels, not color alone. The numeric table remains meaningful when the plot is unavailable or motion is reduced.

The plotted angle is an aid to reasoning, not a separate definition. When the controls change, the numerical table recomputes the same formula shown above. A pair that becomes longer without changing direction keeps the cosine value, while rotating either vector changes both the angle and the score.

Use the score carefully in embedding retrieval

An embedding is a vector representation of an item such as a document, query, image, or user. If the intended match is orientation, cosine similarity is a natural score: it compares normalized vectors rather than their raw lengths. The score still depends on the embedding model, preprocessing, corpus, and task; it does not turn “similar” into a universal property of the underlying documents.

The current scikit-learn pairwise-metrics guide describes cosine similarity as an L2-normalized dot product and notes its use for document vectors. Its cosine_similarity API treats rows as samples and returns a matrix of pairwise scores. That shape is easy to lose when moving between one vector and a batch, so make the row convention explicit:

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

query = np.array([[3.0, 1.0]])       # shape (1 sample, 2 features)
documents = np.array([
    [3.0, 1.0],
    [1.0, 2.0],
    [30.0, 10.0],
])                                    # shape (3 samples, 2 features)

scores = cosine_similarity(query, documents)
print(scores.round(4))
# [[1.     0.7071 1.    ]]

For a transparent NumPy implementation, the one-dimensional dot product and vector norms are enough. NumPy’s dot documentation defines the 1-D case as the inner product of two vectors, while numpy.linalg.norm accepts an axis argument when an array contains multiple vectors. The guard is part of the implementation rather than a cosmetic check:

import numpy as np

def cosine_similarity(x, y):
    x = np.asarray(x, dtype=float)
    y = np.asarray(y, dtype=float)
    x_norm = np.linalg.norm(x)
    y_norm = np.linalg.norm(y)
    if x_norm == 0 or y_norm == 0:
        raise ValueError('cosine similarity needs two nonzero vectors')

    raw = np.dot(x, y) / (x_norm * y_norm)
    return np.clip(raw, -1.0, 1.0)

print(cosine_similarity(np.array([3.0, 1.0]), np.array([1.0, 2.0])))
# 0.7071067811865475

The clip keeps a tiny floating-point overshoot from sending arccos outside its domain. It should not be used to hide a larger numerical or data-shape error. For a batch with shape (number_of_samples, number_of_features), compute a norm across axis=1 and keep the resulting one score per row; the current NumPy documentation demonstrates this axis convention for vector norms.

Boundaries that change the interpretation

A zero vector has no angle

If $x=0$ , then $\lVert x\rVert_2=0$ and the cosine formula divides by zero. The dot product $0^\mathsf{T}y=0$ is still defined, but it does not mean that the zero vector points at a right angle to $y$ . A zero vector has no direction, so the angle and cosine similarity are undefined mathematically.

Libraries may choose a policy for this input. The current cosine_similarity example includes a zero row and displays a score of 0.0; that is an API convention to check for the version and data path you use, not a newly defined direction. A production retrieval pipeline should decide whether to reject, filter, or separately route zero embeddings and test that policy explicitly.

Opposite vectors are not “unrelated”

For nonzero $x$ , the pair $(x,-x)$ has

\cos(x,-x)=\frac{-\lVert x\rVert_2^2}{\lVert x\rVert_2^2}=-1, \qquad \theta=\pi.

The score is maximally negative because the vectors point in opposite directions. A score near $0$ means orthogonality under this geometry; it does not automatically mean that two items are semantically unrelated. The interpretation belongs to the representation and task.

Cosine is not the same as distance or a raw dot product

Cosine similarity ignores positive scale. Euclidean distance does not. For $x=(1,1)$ and $y=(10,10)$ , cosine similarity is $1$ , but

\lVert x-y\rVert_2=\sqrt{(-9)^2+(-9)^2}=9\sqrt{2}\approx12.728.

The pair is directionally identical and still far apart as points. Conversely, a raw dot product mixes direction with both lengths. Choose cosine when directional agreement is the intended signal; choose a norm-induced distance or a raw score when magnitude should remain part of the decision. The scikit-learn cosine_distances API defines its value as $1$ minus cosine similarity, so it preserves the same directional choice rather than recovering Euclidean separation.

The next lesson, Lesson 045: Orthogonality means no linear overlap, focuses on the special case where the inner product is zero and explains what “no linear overlap” means. The Math for AI course hub keeps the dependency sequence visible as each lesson becomes available.

Check your understanding

Question 1

For x = (3, 1) and y = (1, 2), compute the dot product, both L2 norms, cosine similarity, and angle.