Linear Independence Removes Redundancy from AI Features

A model can receive two feature columns that look like separate inputs but always rise and fall together. If one column is an exact multiple of the other, the model has been given two names for one direction. The prediction may still be well defined, while the individual weights that produced it are not.

Linear independence gives us a precise test for that redundancy. By the end of this lesson, you will be able to find a nonzero coefficient recipe that adds vectors to zero, explain why that recipe exposes a repeated direction, and check a small AI feature matrix in NumPy.

Lesson 17 introduced linear combinations as weighted sums. Lesson 28 used those combinations to describe restricted features. Section 2.5 of the official Mathematics for Machine Learning uses the same operation to decide whether a collection of vectors contains a redundant direction. The authors’ companion website identifies the linked PDF as the current book download. The feature matrix, worked arithmetic, visual, code, and exercises here are original.

Look for a nonzero recipe that cancels

Suppose we have vectors $\boldsymbol{x}_1,\ldots,\boldsymbol{x}_k$ in one vector space. We ask whether some coefficients can combine them into the zero vector:

\lambda_1\boldsymbol{x}_1+\cdots+\lambda_k\boldsymbol{x}_k=\boldsymbol{0}.

The all-zero coefficients always work. The useful question is whether any other choice works.

The zero result alone proves nothing, because setting every coefficient to zero always produces it. Dependence means there is a nontrivial recipe: at least one coefficient is nonzero, yet the weighted directions cancel. One vector in the collection can then be rewritten using the others, so the collection carries a repeated linear direction.

Work the cancellation through

Take two feature directions:

\boldsymbol{u}= \begin{bmatrix}1\\2\end{bmatrix}, \qquad \boldsymbol{v}= \begin{bmatrix}2\\4\end{bmatrix}.

The second vector doubles the first: $\boldsymbol{v}=2\boldsymbol{u}$ . Choose coefficients $2$ and $-1$ :

2\boldsymbol{u}-\boldsymbol{v} =2\begin{bmatrix}1\\2\end{bmatrix} -\begin{bmatrix}2\\4\end{bmatrix} =\begin{bmatrix}2-2\\4-4\end{bmatrix} =\begin{bmatrix}0\\0\end{bmatrix}.

This is a nontrivial relation because the coefficients are not both zero. It also gives a direct rewrite: $\boldsymbol{v}=2\boldsymbol{u}$ . Either vector can be removed without changing the set of directions reachable by linear combinations.

The coefficient search below uses $\boldsymbol{u}=(1,2)$ and, in its dependent setting, $\boldsymbol{v}=(2,4)$ . Move the coefficients to see the scaled vectors add. Switch the second vector to $(0,1)$ to compare an independent pair. The text result and table repeat the plotted coordinates, so the calculation remains available without interpreting color or shape.

Search for a nontrivial coefficient pair whose sum is the zero vector.

Choose the vector pairCollinear pair: u = (1, 2), v = (2, 4)Independent pair: u = (1, 2), v = (0, 1)

Set the coefficients

Use the arrow keys while a slider is focused. Each step changes its coefficient by 0.25.

Coefficient a for vector u+1Coefficient b for vector v-0.5

For a = +1 and b = -0.5, a u + b v = (0, 0). This is a nontrivial zero combination, so this pair is dependent.

a u contribution
b v contribution
sum a u + b v

Current coefficient search values
Quantity	Coordinates or value
Coefficient a	+1
Coefficient b	-0.5
Vector u	(+1, +2)
Vector v	(+2, +4)
a u	(+1, +2)
b v	(-1, -2)
Sum a u + b v	(0, 0) — zero vector

The sliders search only coefficients from −1 to 1. A plotted cancellation demonstrates dependence; a finite search by itself cannot prove independence.

The independent comparison makes the definition concrete. Let $\boldsymbol{p}=(1,2)$ and $\boldsymbol{q}=(0,1)$ . If $a\boldsymbol{p}+b\boldsymbol{q}=\boldsymbol{0}$ , then

a\begin{bmatrix}1\\2\end{bmatrix} +b\begin{bmatrix}0\\1\end{bmatrix} =\begin{bmatrix}a\\2a+b\end{bmatrix} =\begin{bmatrix}0\\0\end{bmatrix}.

The first coordinate forces $a=0$ . Substituting that into the second gives $b=0$ . Since only the trivial coefficients work, the pair is independent. This algebra is the proof; the interactive grid is a way to explore what different coefficient choices do.

Find repeated feature directions in a model

A design matrix stores examples in rows and features in columns. Suppose a three-example model receives the columns

\boldsymbol{x}_1= \begin{bmatrix}1\\2\\3\end{bmatrix}, \qquad \boldsymbol{x}_2= \begin{bmatrix}2\\4\\6\end{bmatrix} =2\boldsymbol{x}_1, \qquad X= \begin{bmatrix} 1&2\\ 2&4\\ 3&6 \end{bmatrix}.

The columns are dependent because $-2\boldsymbol{x}_1+\boldsymbol{x}_2=\boldsymbol{0}$ . For a linear model with weights $w_1,w_2$ , the predictions are

X\begin{bmatrix}w_1\\w_2\end{bmatrix} =\boldsymbol{x}_1w_1+\boldsymbol{x}_2w_2 =\boldsymbol{x}_1(w_1+2w_2).

Only the combined weight $w_1+2w_2$ affects these predictions. For example, weights $(1,0)$ and $(-1,1)$ both make that quantity equal to $1$ , so both produce $(1,2,3)^\mathsf{T}$ . The prediction is the same, but the individual weights are not uniquely determined by these data. This is one concrete way feature redundancy can complicate interpretation.

NumPy’s matrix_rank reports the number of singular values above its tolerance. That gives a practical rank check for floating-point matrices:

import numpy as np

X = np.array([[1., 2.],
              [2., 4.],
              [3., 6.]])

print(X @ np.array([-2., 1.]))  # [0. 0. 0.]
print(np.linalg.matrix_rank(X)) # 1

The first line checks the exhibited relation, while the second reports one independent column direction. The NumPy matrix_rank documentation explains that the calculation uses singular values and a tolerance. Floating point roundoff means an exact zero singular value may appear as a tiny nonzero number, so software applies a threshold.

If two measured columns are almost multiples rather than exact multiples, the mathematical test and a numerical rank decision need care. NumPy notes that its default tolerance can treat sufficiently close columns as rank deficient and that measurement uncertainty may justify a different threshold. For a least-squares fit, the NumPy least-squares documentation also describes rank determination through a singular-value cutoff and returns a minimum-norm solution when multiple solutions minimize the error. A numerical rank is therefore a tolerance-based judgment about the data and computation, not a claim that two measured features are symbolically identical.

The most useful quick checks follow directly from the definition. A zero vector makes a collection dependent. A nonzero scalar multiple duplicates a direction. In $\mathbb{R}^d$ , any collection of more than $d$ vectors must be dependent because there are only $d$ independent directions available. Distinct vectors can still be dependent, as $(1,2)$ and $(2,4)$ demonstrate.

For a small exact example, solve the coefficient equation by hand. For a larger data matrix or decimal measurements, inspect its numerical rank and state the tolerance or uncertainty that informed the interpretation. Once independent directions have been identified, the next lesson asks what all their possible linear combinations can express: the span.

Check your understanding

Question 1

In the definition of linear independence, why do we require that only the all-zero coefficients produce the zero vector?