Inner Products Measure Alignment in Attention

When a Transformer compares a token’s query with another token’s key, the calculation is a dot product. That one number helps decide how much information to collect from the key’s value. The sign can tell us whether two real vectors point broadly together or apart, while the magnitude also depends on their lengths. Keeping those two effects separate makes attention scores easier to read.

By the end, you will be able to compute an inner product, interpret its sign and magnitude under the usual Euclidean geometry, and trace a query-key score through scaling and softmax.

Learning outcome: Compute an inner product and connect its sign and magnitude to alignment.

This lesson builds on weighted sums of vectors from Lesson 017. Section 3.2 of Mathematics for Machine Learning introduces inner products as a way to reason about length, angle, and orthogonality. The numerical examples, attention walkthrough, visual, and assessments here are original.

Multiply matching coordinates to measure alignment

For two real vectors with the same number of coordinates, $x=(x_1,\ldots,x_n)$ and $y=(y_1,\ldots,y_n)$, their dot product is

Pair the first coordinates, multiply them, then add the products from every coordinate. The result is a scalar. A dot product is one kind of inner product; more generally, an inner product is a real-valued rule on pairs of vectors that is linear in each input, symmetric, and positive for every nonzero vector paired with itself. Here we use the standard dot product on $\mathbb{R}^n$.

For a concrete calculation, take $q=(2,1)$ and $k=(1,2)$. Then

The positive score fits the picture: the vectors point into the same half of the plane. Their Euclidean lengths are both $\sqrt5$. Because the two vectors are nonzero and the geometry is the ordinary real Euclidean one, the angle $\theta$ between them satisfies

$$\langle q,k\rangle = \|q\|_2\,\|k\|_2\cos\theta, \qquad \cos\theta = \frac{4}{\sqrt5\sqrt5}=0.8.$$

So the dot product is not just a direction comparison. It combines direction with both vector lengths. The cosine value $0.8$ isolates directional alignment by dividing out those lengths; that normalized quantity is called cosine similarity and belongs to a later lesson.

• Query vector: q = (2, 1)
• Key direction: k = (1, 2)
• Projection vector: projₖ(q)
• Perpendicular remainder: q − projₖ(q)

q · k = 2(1) + 1(2) = 4

projₖ(q) = (4/5)k = (0.8, 1.6)

Signed scalar projection: 4/√5 ≈ 1.79

A projection shows which part points along another vector

Projecting $q$ onto the direction of a nonzero vector $k$ asks: how much of $q$ lies along $k$? Its signed scalar length is

$$\operatorname{comp}_{k}(q)=\frac{\langle q,k\rangle}{\|k\|_2}.$$

The sign is retained: a negative result means the component points against $k$. For our example, $\operatorname{comp}_{k}(q)=4/\sqrt5\approx1.79$. The projected vector is that scalar length times the unit vector in the direction of $k$:

$$\operatorname{proj}_{k}(q) =\frac{\langle q,k\rangle}{\|k\|_2^2}k =\frac45(1,2)=(0.8,1.6).$$

The leftover vector is $q-\operatorname{proj}_{k}(q)=(1.2,-0.6)$. Check that it is perpendicular to $k$:

$$(1.2,-0.6)\cdot(1,2)=1.2-1.2=0.$$

This decomposition explains the geometry: the dot product measures the signed amount along $k$ after accounting for $k$‘s own length, while the perpendicular remainder contributes nothing to that product.

For nonzero real vectors in standard Euclidean coordinates, a positive inner product means an angle below $90^\circ$, zero means perpendicular vectors, and a negative product means an angle above $90^\circ$. The zero vector is a boundary case: its dot product with every vector is zero, but it has no direction, so there is no angle to interpret. More generally, these angle statements assume the usual real Euclidean inner product. A different inner product can define a different geometry.

Attention uses the score before turning it into a weight

In attention, a query describes what a position is looking for, while each key describes what another position can match against. A learned projection produces these query and key vectors from token representations. The original Transformer paper describes their compatibility as a dot product, then divides each score by $\sqrt{d_k}$, where $d_k$ is the number of coordinates in each query and key. A softmax across the allowed keys turns the scaled scores into weights on the corresponding value vectors (Vaswani et al., Attention Is All You Need).

For one query $q$ and a matrix of keys $K$, the scaled scores are $qK^\mathsf{T}/\sqrt{d_k}$. If there is a single query, each entry is one query-key score. With multiple queries, the matrix product computes many such scores together. Softmax is then applied across the key positions for each query, and those weights form a weighted sum of the values.

Try a tiny example with $q=(1,2)$, $k_1=(1,0)$, and $k_2=(0,1)$. Both keys have dimension $d_k=2$:

The dot products $1$ and $2$ are raw compatibility scores. After scaling, $0.707$ and $1.414$ are logits. The values $0.330$ and $0.670$ are softmax weights; with no dropout, they sum to one over the two unmasked keys. These are three related but different quantities. The weights form a normalized distribution for this query, but they are not calibrated chances that each key is semantically relevant. PyTorch’s scaled_dot_product_attention follows this sequence: multiply queries by transposed keys, apply the scale, add any mask, apply softmax, then multiply the weights by the values. Its default scale is $1/\sqrt{d_k}$.

The scale is not a cosine normalization. It divides by a factor based on the dimension, not by each query’s and key’s individual lengths. The Transformer paper motivates this scaling by noting that, under its stated assumption of independent zero-mean unit-variance query and key coordinates, the unscaled dot product has variance $d_k$. Dividing by $\sqrt{d_k}$ keeps score spread from growing with the dimension under that assumption. It does not force each query and key to have unit length or make the scores cosine similarities.

Keep score size separate from directional alignment

A large positive dot product can arise from a close alignment, long vectors, or both. For example, $(4,0)\cdot(1,0)=4$ has cosine similarity $1$, while $(2,1)\cdot(1,2)=4$ has cosine similarity $0.8$. The same raw score does not imply the same angle. Likewise, multiplying one vector by a positive scalar increases the dot product without changing its direction; multiplying it by $-1$ flips the sign and reverses its direction.

In an attention head, query and key lengths are learned as part of the model’s representations. The score is used as a compatibility logit, then the softmax compares it with other available scores for that query. It is not a standalone probability or a universal measure of semantic similarity. If a system instead normalizes each nonzero vector to unit length, it changes the score to cosine similarity and removes individual vector magnitudes. That can be useful for tasks such as comparing document embeddings, but it is a different operation from the standard scaled dot-product attention above.

You can verify the arithmetic with NumPy:

import numpy as np

q = np.array([2.0, 1.0])
k = np.array([1.0, 2.0])

dot = np.dot(q, k)
scalar_projection = dot / np.linalg.norm(k)
vector_projection = dot / np.dot(k, k) * k
cosine_similarity = dot / (np.linalg.norm(q) * np.linalg.norm(k))

print(dot)                 # 4.0
print(scalar_projection)  # 1.7888543819998317
print(vector_projection)  # [0.8 1.6]
print(cosine_similarity) # 0.8

The same multiplication can now be read in two settings: as a geometric measure of signed alignment, and as the compatibility score used to compare a query with keys. The next step is to read a dot product as a feature-weighted score, where each coordinate contributes its own signed amount.

Check your understanding

Question 1

Show the step-by-step solution

Compute $(2,-1,3)\cdot(1,4,-2)$. What does its sign say about the angle between these nonzero vectors in standard real Euclidean space?

First multiply matching coordinates and add:

$$2\cdot1 + (-1)\cdot4 + 3\cdot(-2)=2-4-6=-8.$$

The dot product is negative. For two nonzero real vectors under the standard Euclidean inner product, a negative product means their angle is greater than $90^\circ$. It does not identify the exact angle without also using their lengths.

Question 2

Show the step-by-step solution

For $u=(2,1)$ and $v=(1,2)$, calculate $u\cdot v$ and the cosine of the angle between them.

The product is

$$u\cdot v=2\cdot1+1\cdot2=4.$$

Their lengths are both $\sqrt5$, so the Euclidean angle formula gives

$$\cos\theta=\frac{u\cdot v}{\|u\|_2\|v\|_2} =\frac{4}{\sqrt5\sqrt5}=\frac45=0.8.$$

The dot product is $4$; the cosine similarity is $0.8$. They are not the same quantity.

Question 3

Show the step-by-step solution

Project $q=(2,1)$ onto $k=(1,2)$. Find both the signed scalar length and the projection vector.

First, $q\cdot k=4$ and $\|k\|_2=\sqrt5$. The signed scalar projection is

$$\operatorname{comp}_{k}(q)=\frac{4}{\sqrt5}\approx1.79.$$

For the vector projection, divide the dot product by $\|k\|_2^2=5$ and multiply by $k$:

$$\operatorname{proj}_{k}(q)=\frac45(1,2)=(0.8,1.6).$$

Question 4

Show the step-by-step solution

Two nonzero real vectors have standard dot product zero. What can you conclude, and what extra caution is needed if one vector is zero?

For nonzero vectors in standard Euclidean space, $x\cdot y=\|x\|_2\|y\|_2\cos\theta$. Since both lengths are positive, a zero product forces $\cos\theta=0$, so the vectors are perpendicular. If one vector is zero, the product is also zero, but the zero vector has no direction and no angle with the other vector.

Question 5

Show the step-by-step solution

Both $(4,0)\cdot(1,0)$ and $(2,1)\cdot(1,2)$ equal $4$. Why do the equal products not imply equal directional alignment?

The first pair points in exactly the same direction, so its cosine similarity is $1$. In the second pair, each length is $\sqrt5$, so its cosine similarity is $4/(\sqrt5\sqrt5)=0.8$. Both products happen to equal $4$, but the vector lengths differ across the two pairs. A raw dot product combines length and direction, so it cannot by itself identify the angle.

Question 6

Show the step-by-step solution

For $q=(1,2)$, $k_1=(1,0)$, and $k_2=(0,1)$, compute the raw scores, scaled logits, and softmax weights for $d_k=2$. Which key gets the larger weight?

The dot products are $q\cdot k_1=1$ and $q\cdot k_2=2$. Dividing by $\sqrt{d_k}=\sqrt2$ gives logits about $0.707$ and $1.414$. Applying softmax gives

$$\left(\frac{e^{0.707}}{e^{0.707}+e^{1.414}}, \frac{e^{1.414}}{e^{0.707}+e^{1.414}}\right) \approx(0.330,0.670).$$

Key $k_2$ receives the larger weight. The softmax weights sum to one across these two unmasked positions, while the dot products and scaled logits are scores rather than probabilities.

Question 7

Show the step-by-step solution

Suppose an attention query is the zero vector and two keys are unmasked. Under the usual softmax calculation with dropout disabled, what weights result? Is the angle between the query and either key defined?

The zero query has dot product zero with each key. Scaling keeps both logits at zero, and

$$\operatorname{softmax}(0,0)=\left(\frac12,\frac12\right).$$

The two keys therefore receive equal weights. The angle is undefined because the zero query has no direction. Equal scores here do not mean the query is geometrically aligned with either key.

Question 8

Show the step-by-step solution

An implementation replaces each query-key dot product with cosine similarity before softmax. What changes for $q=(2,1)$ and $k=(1,2)$, and why is this not the same as the standard attention scale?

The raw dot product is $4$. The two lengths are $\sqrt5$, so cosine similarity is $4/(\sqrt5\sqrt5)=0.8$. Cosine normalization divides by each vector’s length and removes their magnitude from the score. Standard scaled dot-product attention instead divides by $\sqrt{d_k}$, a factor shared by all vectors of that head’s dimension. It keeps magnitude effects in the logits and is not cosine similarity.

Sources

1. Mathematics for Machine Learning book PDF, Section 3.2
2. Attention Is All You Need
3. PyTorch documentation: scaled_dot_product_attention