Span: The Predictions a Linear Model Can Reach

A linear prediction rule can adjust its weights as much as it likes and still miss a pattern its features cannot express. The limit is visible across the training examples: every prediction vector is a weighted sum of the feature columns. Those weighted sums form a set called the span.

By the end of this lesson, you will be able to describe that reachable set, calculate a prediction inside it, and test whether a proposed set of target values could be produced by the model at all.

Learning outcome: Identify the set of vectors reachable by linear combinations.

Lesson 29: Linear independence removes redundancy asks whether feature directions add distinct information. Here we ask the next question: what combinations of those directions can a model actually reach? Section 2.5 of the official Mathematics for Machine Learning book introduces linear combinations and linear independence, the vocabulary needed for this lesson’s definition of span. The prediction example, calculations, code, visual, and assessments here are original.

A model’s outputs live in a reachable set

Suppose a dataset has $n$ examples and $p$ numeric features. Put feature $j$ ‘s values for all $n$ examples into a column vector $\boldsymbol{x}_j\in\mathbb{R}^n$ . The design matrix $X\in\mathbb{R}^{n\times p}$ stacks these columns. For a weight vector $\boldsymbol{w}=(w_1,\ldots,w_p)^\mathsf{T}\in\mathbb{R}^p$ , an intercept-free linear model predicts

\hat{\boldsymbol{y}}=X\boldsymbol{w} =w_1\boldsymbol{x}_1+\cdots+w_p\boldsymbol{x}_p.

The matrix-vector product is doing something conceptually important: it mixes the feature columns using the model’s weights. The coefficients can vary, but the only directions available are the columns already in $X$ .

The last sentence is the capacity claim for this finite dataset: the span contains exactly the prediction vectors the unconstrained linear weights can produce. It does not claim the model can make every possible mapping on future inputs. That depends on the feature rule, its parameters, and what inputs it will see.

Work out the reachable predictions

Consider three examples and two features:

X=\begin{bmatrix} 1&0\\ 0&1\\ 1&1 \end{bmatrix}, \qquad \boldsymbol{x}_1=\begin{bmatrix}1\\0\\1\end{bmatrix}, \quad \boldsymbol{x}_2=\begin{bmatrix}0\\1\\1\end{bmatrix}.

Each weight pair $(a,b)$ produces

a\boldsymbol{x}_1+b\boldsymbol{x}_2 =a\begin{bmatrix}1\\0\\1\end{bmatrix} +b\begin{bmatrix}0\\1\\1\end{bmatrix} =\begin{bmatrix}a\\b\\a+b\end{bmatrix}.

So every reachable prediction vector has the form $[a,b,a+b]^\mathsf{T}$ . Its third coordinate must equal the sum of the first two. This is a plane through zero inside $\mathbb{R}^3$ : three output coordinates are recorded, but the two feature directions can only fill that plane.

Choose weights, then test a target

Take weights $\boldsymbol{w}=[2,-1]^\mathsf{T}$ . Multiply each feature column by its weight, then add:

X\boldsymbol{w} =2\begin{bmatrix}1\\0\\1\end{bmatrix} -\begin{bmatrix}0\\1\\1\end{bmatrix} =\begin{bmatrix}2\\0\\2\end{bmatrix} -\begin{bmatrix}0\\1\\1\end{bmatrix} =\begin{bmatrix}2\\-1\\1\end{bmatrix}.

The result satisfies the plane rule: $1=2+(-1)$ . Now ask whether target $\boldsymbol{y}=[1,2,4]^\mathsf{T}$ can be produced. Its first two entries would force $a=1$ and $b=2$ . Those weights would make the third entry $a+b=3$ , not $4$ . No weight pair can produce this target with the two given features.

The failure is a feature limit, not a failure to search long enough for the right coefficients. A model optimizer can choose different weights, but it cannot leave the column span without changing the feature directions or the model form.

Prediction space

Feature columns fence in every prediction

With two feature columns, the model can reach the plane they span. The target vector sits outside because its third value is not the sum of the first two.

The plane represents all vectors [a, b, a + b]. The finite plane patch is only a viewing window; the span continues without bound in every in-plane direction. The dashed target violates the coordinate rule.

Feature combinations and their membership in the column span
Vector	Coordinates	Span check
Feature column x₁	[1, 0, 1]	Inside: third = first + second
Feature column x₂	[0, 1, 1]	Inside: third = first + second
Prediction for w = [2, −1]	[2, −1, 1]	Inside: third = first + second
Proposed target	[1, 2, 4]	Outside: 4 ≠ 1 + 2

The diagram shows the same algebra from the model’s point of view. The solid feature arrows generate the plane, and any weighted combination of them stays there. The dashed target lies off it. The visual makes the location intuitive; the equation $y_3=y_1+y_2$ is the exact test.

Check the limit with a small NumPy calculation

The model computation follows the definition directly. NumPy’s matmul implements the matrix-vector product, so multiplying $X$ by the weights gives the prediction vector:

import numpy as np

X = np.array([[1, 0], [0, 1], [1, 1]])
weights = np.array([2, -1])
prediction = X @ weights
target = np.array([1, 2, 4])

print(prediction)  # [ 2 -1  1]
print(prediction[2] == prediction[0] + prediction[1])  # True
print(target[2] == target[0] + target[1])  # False

The first line confirms the worked prediction. The second checks the rule that all vectors in this column span must satisfy. The third rejects the proposed target using the same exact relation. This is a structural check; it does not depend on how the weights were trained.

This view also explains a familiar linear-regression constraint. The scikit-learn linear-model guide defines a prediction as a weighted sum of features, with an optional intercept. The fit procedure may try to reduce errors, but on the examples in this lesson an intercept-free model still has only the predictions in $\operatorname{Col}(X)$ . If every feature column is redundant, the span may not grow when a new feature is added; if a new column points in a genuinely new direction, it can enlarge the reachable set.

Span is about the available directions

The span includes every real coefficient value, including zero and negative weights. That matters: considering only positive combinations would describe a smaller set, often a cone, rather than the full span. Standard unconstrained linear regression permits positive or negative coefficients. Constraints can limit which coefficients are allowed, while regularization changes which weights a training objective prefers; neither changes the column span defined by the feature matrix itself.

The number of feature columns alone does not tell us how many independent directions they provide. Two columns may point along the same line, in which case their span is still one-dimensional. The next step is to determine when a new vector adds an independent direction and can expand the span. That is the question addressed by Lesson 29: linear independence removes redundancy.

The practical habit is simple: when a model repeatedly misses a pattern, ask whether its features can express the needed output before blaming the optimizer. The span tells you which prediction vectors are even on the table.