Subspaces for AI: Test Restricted Feature Families

An encoder may produce hundreds of coordinates for every input even when the nearby representations move along only a few coherent directions. To reason about that restriction, we need more than a cluster of points. We need a set that remains valid when its directions are added and scaled.

That set is a vector subspace. By the end of this lesson, you will be able to run the subspace test, prove that a plane through the origin passes it, reject tempting counterexamples, and interpret a lower-dimensional linear model of learned representation offsets.

Learning outcome: Test subspace conditions and interpret a lower-dimensional region.

Lesson 27: Vector spaces define legal representations established the ambient rules for vector addition and scalar multiplication. A subspace keeps those same operations and narrows the allowed vectors. The official Mathematics for Machine Learning companion develops vector subspaces in Section 2.4.3 of the book PDF. The representation scenario, calculations, code, visual, and assessments below are original.

A restriction must survive the operations

Let the ambient vector space be $V=\mathbb{R}^3$ , and let $U\subseteq V$ be a candidate set. Calling $U$ a subspace means that vectors can stay inside $U$ while using the addition and scalar multiplication already defined on $V$ . The ambient space supplies the arithmetic; the subset has to prove that it is closed under that arithmetic.

The zero-vector check is fast and decisive. Any subspace must contain zero because scalar closure requires $0u=0$ for every $u\in U$ . Addition then says that two allowed directions can be combined. Scalar closure says that an allowed direction remains allowed when it is stretched, shrunk, reversed, or collapsed to zero.

The compact linear-combination test packages those requirements together. Set $\alpha=\beta=1$ to recover addition, set $\beta=0$ to recover scaling, and set both coefficients to zero to recover the zero vector. This is the same linear-combination operation introduced in Lesson 17, now used as a membership test for an entire family of vectors.

Worked example: a plane through the origin

Consider the set

U=\left\{ \begin{bmatrix}x_1\\x_2\\x_3\end{bmatrix}\in\mathbb{R}^3 :x_3=x_1+x_2 \right\}.

The equation restricts three displayed coordinates using one relation. Choose $x_1=a$ and $x_2=b$ freely; then $x_3=a+b$ . Every vector in $U$ therefore has the form

\begin{bmatrix}a\\b\\a+b\end{bmatrix} =a\begin{bmatrix}1\\0\\1\end{bmatrix} +b\begin{bmatrix}0\\1\\1\end{bmatrix}.

Two numbers, $a$ and $b$ , determine all three coordinates. Geometrically, the result is a plane inside three-dimensional space. The plane passes through the origin because $a=b=0$ gives $(0,0,0)$ .

Now perform every part of the test. Write two arbitrary members as

u=\begin{bmatrix}a\\b\\a+b\end{bmatrix}, \qquad v=\begin{bmatrix}c\\d\\c+d\end{bmatrix}.

Their sum is

u+v =\begin{bmatrix}a+c\\b+d\\a+b+c+d\end{bmatrix}.

The third coordinate equals the first plus the second: $(a+c)+(b+d)=a+b+c+d$ . Therefore $u+v\in U$ . Scaling $u$ by any real $\lambda$ gives

\lambda u =\begin{bmatrix}\lambda a\\\lambda b\\\lambda(a+b)\end{bmatrix} =\begin{bmatrix}\lambda a\\\lambda b\\\lambda a+\lambda b\end{bmatrix},

which also satisfies the defining equation. Together with the zero-vector check, this proves that $U$ is a subspace of $\mathbb{R}^3$ .

Plane-in-space map

A subspace keeps every linear combination on the plane

The blue plane contains the origin and every vector whose third coordinate equals the sum of the first two. Solid arrows stay on the plane; the dashed arrow fails the equation.

The plane needs two freely chosen coordinates even though its vectors are written with three. Passing through zero and remaining closed under addition and scaling make it a subspace.

Coordinate checks represented in the plane diagram
Vector	Coordinates	Membership test	Result
u	[1, 0, 1]	1 = 1 + 0	Inside U
v	[0, 1, 1]	1 = 0 + 1	Inside U
u − v	[1, −1, 0]	0 = 1 + (−1)	Inside U
r	[1, 0, 0]	0 ≠ 1 + 0	Outside U

The picture suggests the result, while the algebra proves it: a drawing can show a few arrows, but closure must hold for every pair of vectors and every real coefficient.

The diagram answers the geometric question: “Where can these features move?” The plane is lower-dimensional than its ambient space because one coordinate is forced by the other two. Still, the proof matters more than the perspective drawing. A few arrows landing on a plane cannot establish closure for infinitely many possible vectors.

Three attractive sets that fail

Subspaces are special among subsets. Shape, symmetry, and visual flatness are not sufficient.

First, shift the worked plane by one unit:

W=\{x\in\mathbb{R}^3:x_3=x_1+x_2+1\}.

This is still a flat plane, but $(0,0,0)$ fails because $0\neq0+0+1$ . Therefore $W$ is not a vector subspace. Scaling also exposes the problem: if $w\in W$ , then $2w$ generally satisfies $x_3=x_1+x_2+2$ , not the equation defining $W$ .

Second, unit-length vectors do not form a subspace. A normalized embedding $e$ may satisfy $\lVert e\rVert=1$ , while $2e$ has length $2$ and $0e$ has length $0$ . The set fails scalar closure and excludes the zero vector. This is why the normalized embeddings discussed in Lesson 27 can be useful model inputs without forming a vector space or subspace.

Third, a finite dataset is usually not a subspace. Even if every recorded representation satisfies $x_3=x_1+x_2$ , the stored rows alone need not include their sums, negatives, or every real scaling. The constraint-defined family $U$ is a subspace; the finite sample used to discover or estimate it is merely a subset of $U$ .

NumPy can check examples, not replace the proof

For the worked plane, store the two generating directions as rows. Each row of coefficients tells NumPy how much of each direction to combine:

import numpy as np

directions = np.array([
    [1.0, 0.0, 1.0],
    [0.0, 1.0, 1.0],
])

coefficients = np.array([
    [2.0, -1.0],
    [-1.0, 3.0],
    [0.5, 0.25],
])

offsets = coefficients @ directions
constraint = np.array([1.0, 1.0, -1.0])
constraint_residuals = offsets @ constraint

print(offsets)
print(constraint_residuals)
print(np.allclose(
    constraint_residuals,
    0.0,
    rtol=0.0,
    atol=1e-12,
))

The output is

[[ 2.   -1.    1.  ]
 [-1.    3.    2.  ]
 [ 0.5   0.25  0.75]]
[0. 0. 0.]
True

The first @ forms three linear combinations. The second evaluates $x_1+x_2-x_3$ for each result. The NumPy matmul documentation defines the shape rule implemented by @; here, a $3\times2$ coefficient matrix times a $2\times3$ direction matrix produces three vectors in $\mathbb{R}^3$ .

The exact example produces zeros, but real model activations use finite-precision arithmetic. The NumPy allclose documentation defines the tolerance comparison and warns that its default absolute tolerance can be inappropriate for values much smaller than one. Setting rtol=0.0 and choosing an explicit atol makes the intended zero test visible. The tolerance still has to match the activation scale and the error that the application can accept; 1e-12 is appropriate for this constructed float64 example, not a universal threshold.

This program checks chosen vectors against a known constraint. It does not prove that an arbitrary cloud of model activations belongs to the plane. That broader claim needs a fitted model, residual analysis, and evidence that the chosen linear approximation is adequate for the region being studied.

Representation manifolds are usually only locally flat

Let an encoder map an input $x$ to a representation $h(x)\in\mathbb{R}^d$ . Choose an anchor representation $h_0$ and examine nearby offsets

\Delta h=h(x)-h_0.

If those offsets are well approximated by combinations of a small collection of directions, a subspace $U$ gives a useful local linear model. In the original coordinates, the approximation is the shifted set

h_0+U=\{h_0+u:u\in U\}.

The offset family $U$ passes through zero. The shifted representation patch $h_0+U$ passes through $h_0$ and is generally not itself a subspace of $\mathbb{R}^d$ . Centering is what separates a legal direction from an absolute location.

This viewpoint is an approximation, not a claim that every learned representation lies on one global plane. In Yoshua Bengio’s primary paper on deep representations, manifold learning is described as finding low-dimensional high-density regions and coordinate systems for them. The paper presents PCA as a linear ancestor of manifold learning and also explains why a global linear assumption can miss curved structure. Locally, a plane can summarize the directions that vary near one anchor; farther away, the useful directions may rotate or the approximation error may grow.

For an AI practitioner, that creates a concrete interpretation rule:

a subspace describes a family of allowed offsets closed under mixing and scaling;
a shifted plane describes possible representations near an anchor; and
a curved representation manifold may need different local planes in different neighborhoods.

The subspace test tells us whether the linear object is internally consistent. It does not tell us whether the object fits the data well, captures a causal feature, or preserves the information a downstream task needs.

Boundary cases sharpen the test

Two subspaces are easy to overlook. The set containing only the zero vector, $\{0\}$ , passes every condition: zero plus zero is zero, and every scalar times zero is zero. The whole ambient space $V$ is also a subspace of itself. These are called trivial subspaces, although both are useful boundary cases when debugging an argument.

The empty set fails because it does not contain zero. A single nonzero vector $\{u\}$ also fails over the real numbers: $2u$ is not in the set unless $u=0$ . By contrast, the entire line $\{\lambda u:\lambda\in\mathbb{R}\}$ does pass because it includes every real scaling, their sums, and zero.

Subspaces give precise language for restricted feature families. Once a set passes the test, any linear combination stays legal. The next lesson asks a different question about the directions used to describe that set: whether some of them are redundant.