Vector Spaces: Test Whether AI Representations Stay Legal

An embedding pipeline can force each nonzero output vector to length $1$. That makes similarity calculations convenient, but it creates a subtle trap: the normalized outputs do not form a vector space under ordinary vector addition and scalar multiplication. Add two of them or change one vector’s scale, and the result can leave the allowed set.

This lesson turns “vector space” into a test rather than a label. By the end, you will be able to check the zero vector, addition closure, and scalar closure; construct one decisive counterexample when a rule fails; and explain why normalized embeddings live inside an ambient vector space without themselves forming one.

Learning outcome: Test whether a set satisfies vector-space closure rules.

Lesson 17: Linear combinations build new vectors supplied the operation $a\boldsymbol{x}+b\boldsymbol{y}$. Lesson 26: Groups, rings, and fields explain valid operations showed why a set and its operations must be considered together. Section 2.4 of the official Mathematics for Machine Learning book formalizes that combination as a vector space. The tests, embedding counterexample, code, visual, and assessments below are original.

A vector space promises that linear combinations stay inside

Let $V$ be a set whose elements are called vectors. For a real vector space, two operations are available:

$$+:V\times V\to V, \qquad \cdot:\mathbb{R}\times V\to V.$$

The first operation adds two vectors. The second multiplies a vector by a real number, or scalar. The arrows ending in $V$ encode closure: both operations must return another member of the same set.

Closure alone is not the complete definition. Vector addition must also have a zero vector and additive inverses, addition must be associative and commutative, scalar multiplication must associate correctly, and the familiar distributive and identity rules must hold.

The rule about every scalar is demanding. A set that survives multiplication by $2$ but fails for $-1$, or survives positive scalars but fails for $0$, is not a real vector space.

Inherited operations shorten the test

Most practical questions start with a candidate set $U\subseteq\mathbb{R}^n$ using the ordinary addition and scalar multiplication already defined in $\mathbb{R}^n$. The algebraic laws are inherited from the ambient space. The remaining work is to determine whether the results ever escape $U$.

There is an important asymmetry in how this test is used:

• To prove that $U$ is a vector space, the closure argument must cover every allowed input.
• To prove that $U$ is not a vector space, one valid counterexample is enough.

Checking three convenient vectors cannot prove universal closure. Finding one pair whose sum escapes the set can disprove it immediately.

Worked example: a constraint that survives combinations

Consider the set

$$U=\left\{ \begin{bmatrix}x_1\\x_2\end{bmatrix}\in\mathbb{R}^2: x_1+x_2=0 \right\}.$$

Every vector in $U$ has the form $[t,-t]^\mathsf{T}$. The zero vector belongs to $U$ by choosing $t=0$. Now take arbitrary vectors

$$\boldsymbol{u}=\begin{bmatrix}p\\-p\end{bmatrix}, \qquad \boldsymbol{v}=\begin{bmatrix}q\\-q\end{bmatrix}$$

and arbitrary scalars $a,b\in\mathbb{R}$.

For a numerical check, choose $\boldsymbol{u}=[2,-2]^\mathsf{T}$, $\boldsymbol{v}=[-1,1]^\mathsf{T}$, $a=3$, and $b=-1$:

$$3\boldsymbol{u}-\boldsymbol{v} =\begin{bmatrix}6\\-6\end{bmatrix} +\begin{bmatrix}1\\-1\end{bmatrix} =\begin{bmatrix}7\\-7\end{bmatrix}\in U.$$

The calculation illustrates the rule, while the symbolic argument proves it for all values.

Normalized embeddings fail the same test

Now define the unit-length set

$$S=\left\{\boldsymbol{x}\in\mathbb{R}^d: \lVert\boldsymbol{x}\rVert_2=1\right\}.$$

This is the unit sphere in the ambient vector space $\mathbb{R}^d$. An embedding library may deliberately return vectors in this set. For example, the current Sentence Transformers encode documentation describes a normalize_embeddings option that normalizes returned vectors to length $1$.

In two dimensions, take

$$\boldsymbol{e}_1=\begin{bmatrix}1\\0\end{bmatrix}, \qquad \boldsymbol{e}_2=\begin{bmatrix}0\\1\end{bmatrix}.$$

Both have Euclidean norm $1$, so both are in $S$. Their sum is

$$\boldsymbol{e}_1+\boldsymbol{e}_2 =\begin{bmatrix}1\\1\end{bmatrix}, \qquad \left\lVert\begin{bmatrix}1\\1\end{bmatrix}\right\rVert_2 =\sqrt{1^2+1^2}=\sqrt{2}.$$

Because $\sqrt{2}\ne1$, the sum lies outside $S$. Scalar closure also fails:

$$2\boldsymbol{e}_1=\begin{bmatrix}2\\0\end{bmatrix}, \qquad \lVert2\boldsymbol{e}_1\rVert_2=2.$$

The zero vector supplies a third failure because its norm is $0$, not $1$.

Static closure counterexample

The unit circle rejects ordinary linear combinations

The blue circle contains every two-dimensional vector of Euclidean norm 1. The solid and dashed unit vectors begin inside the set, while their sum, a doubled vector, and the zero vector fail membership.

Exact membership tests represented in the diagram

Closure test	Inputs in S	Result	Result norm	Membership
Addition	e₁ and e₂	e₁ + e₂ = (1, 1)	√2	Outside S
Scalar multiplication	2 and e₁	2e₁ = (2, 0)	2	Outside S
Zero vector	Required additive identity	0 = (0, 0)	0	Not in S

Any one row disproves vector-space membership. The three rows expose different requirements, so fixing one failure would not fix the others.

The visual uses $d=2$ because the geometry fits on a page. The same scalar counterexample works in every dimension: if $\lVert\boldsymbol{x}\rVert_2=1$, then $\lVert2\boldsymbol{x}\rVert_2=2$. Therefore unit-normalized embeddings in $\mathbb{R}^d$ are not a vector space under the usual operations.

Renormalizing changes the operation

A common pipeline adds or averages normalized embeddings, then normalizes the result again. That procedure can be useful, but the final normalization is a new nonlinear operation:

$$\operatorname{normalize}(\boldsymbol{x}+\boldsymbol{y}) =\frac{\boldsymbol{x}+\boldsymbol{y}} {\lVert\boldsymbol{x}+\boldsymbol{y}\rVert_2},$$

provided the denominator is nonzero. The raw sum first leaves the unit sphere; normalization maps it back. Mapping the answer back into $S$ does not make $S$ closed under the original addition operation.

The zero case matters. If $\boldsymbol{y}=-\boldsymbol{x}$, then $\boldsymbol{x}+\boldsymbol{y}=\boldsymbol{0}$, and dividing by its norm would divide by zero. An implementation needs an explicit policy for zero or near-zero rows.

The NumPy linalg.norm documentation specifies the axis and keepdims controls used below to compute one norm per embedding while preserving a broadcastable column shape:

import numpy as np


def normalize_rows(vectors):
    vectors = np.asarray(vectors, dtype=float)
    norms = np.linalg.norm(vectors, axis=1, keepdims=True)
    normalized = np.divide(
        vectors,
        norms,
        out=np.zeros_like(vectors),
        where=norms > 0,
    )
    return normalized


embeddings = np.array([
    [3.0, 4.0],
    [-4.0, 3.0],
    [0.0, 0.0],
])

normalized = normalize_rows(embeddings)
print(normalized)
print(np.linalg.norm(normalized, axis=1))

The output is

[[ 0.6  0.8]
 [-0.8  0.6]
 [ 0.   0. ]]
[1. 1. 0.]

The first two rows land on the unit circle. The zero row remains zero because the function deliberately uses zero as its engineering fallback. That fallback belongs to $\mathbb{R}^2$ but not to $S$, so callers must not claim that every output has unit norm.

A reliable closure review asks about the actual operations

When a model description calls a collection of representations a “space,” identify the set and operations before assuming linear structure. Raw embeddings may live in $\mathbb{R}^d$. Unit-normalized embeddings live on a sphere. Probability vectors with nonnegative coordinates summing to $1$ live in another constrained set. Each can support useful algorithms, but ordinary addition and arbitrary real scaling do not necessarily keep the result legal.

The next lesson studies subspaces: restricted subsets that do pass the closure test and therefore retain the ambient linear structure. Before moving on, use the questions below to practice proving universal closure and finding decisive failures.

Check your understanding

Question 1

For a nonempty subset of Rⁿ with ordinary vector operations, which three membership checks establish that it is a vector space?

Show the step-by-step solution

First check that the zero vector belongs to the set. Second, choose arbitrary $\boldsymbol{x}$ and $\boldsymbol{y}$ in the set and prove that $\boldsymbol{x}+\boldsymbol{y}$ remains in it. Third, choose an arbitrary real scalar $a$ and arbitrary $\boldsymbol{x}$ in the set and prove that $a\boldsymbol{x}$ remains in it. The remaining vector-space laws are inherited from $\mathbb{R}^n$.

Question 2

Show that U = {(t, −t): t ∈ R} is closed under the linear combination a u + b v.

Show the step-by-step solution

Take arbitrary $\boldsymbol{u}=(p,-p)$ and $\boldsymbol{v}=(q,-q)$ in $U$, with arbitrary real $a$ and $b$. Then

$$a\boldsymbol{u}+b\boldsymbol{v} =(ap+bq,-ap-bq).$$

Set $t=ap+bq$. The result is $(t,-t)$, which matches the defining form of $U$. Because the inputs and scalars were arbitrary, this proves closure rather than checking only one example.

Question 3

Use e₁ = (1, 0) and e₂ = (0, 1) to disprove addition closure for unit-normalized vectors.

Show the step-by-step solution

Both inputs have norm $1$, so both belong to the unit-normalized set. Their sum is $(1,1)$. Its norm is $\sqrt{1^2+1^2}=\sqrt{2}$, which is not $1$. Thus two valid members produce a result outside the set. A single such pair disproves addition closure.

Question 4

Does multiplying every unit vector by −1 disprove scalar closure? If not, choose a scalar that does.

Show the step-by-step solution

No. If $\lVert\boldsymbol{x}\rVert_2=1$, then $\lVert-\boldsymbol{x}\rVert_2=1$, so the scalar $-1$ keeps every unit vector on the sphere. Scalar closure must hold for every real scalar, however. Choosing $a=2$ gives $\lVert2\boldsymbol{x}\rVert_2=2$, so $2\boldsymbol{x}$ lies outside the unit sphere and disproves closure.

Question 5

Why must row normalization handle the zero vector explicitly?

Show the step-by-step solution

The usual formula divides $\boldsymbol{x}$ by $\lVert\boldsymbol{x}\rVert_2$. For $\boldsymbol{x}=\boldsymbol{0}$, the norm is $0$, so the formula requests division by zero and is undefined. Code must detect that case and choose a documented policy, such as returning zero, raising an error, or filtering the row. Returning zero is practical in some pipelines, but the result is not unit-normalized.

Question 6

A pipeline averages two unit embeddings and normalizes the average. Why does that not prove the unit sphere is a vector space?

Show the step-by-step solution

First the ordinary average is formed: $\boldsymbol{m}=(\boldsymbol{x}+\boldsymbol{y})/2$. Its norm need not be $1$, so this linear combination can leave the sphere. The pipeline then applies a separate normalization map $\boldsymbol{m}/\lVert\boldsymbol{m}\rVert_2$ to return to the sphere. Closure tests the original addition and scalar multiplication; adding a repair step changes the operation being studied.

Question 7

The nonnegative quadrant Q = {(x₁, x₂): x₁ ≥ 0 and x₂ ≥ 0} contains zero and is closed under addition. Is Q a real vector space?

Show the step-by-step solution

No. Choose $\boldsymbol{x}=(1,0)$, which belongs to $Q$, and the real scalar $a=-1$. Then $a\boldsymbol{x}=(-1,0)$, whose first coordinate is negative, so the result is outside $Q$. The set fails scalar closure even though it contains zero and survives addition.

Sources

1. Mathematics for Machine Learning companion website
2. Mathematics for Machine Learning book PDF
3. NumPy documentation: numpy.linalg.norm
4. Sentence Transformers documentation: SentenceTransformer.encode