Row-Echelon Form Reveals Redundant AI Features

When a model has three feature columns, it is tempting to assume it has three independent directions to learn from. But a feature can be present in the data while adding no new information. If one column is a combination of the others, the model may fit the same predictions with many different parameter vectors.

Row-echelon form makes that hidden structure visible. By the end of this lesson, you will be able to locate pivots, identify free variables, describe a general solution, and use a small NumPy implementation to diagnose redundant features without confusing a numerical warning with a mathematical conclusion.

Learning outcome: Find pivots, free variables, and a general solution.

Lesson 23: Gaussian elimination preserves the solution set introduced the legal row operations that keep a system’s solutions unchanged. Lesson 18: Matrices organize linear rules gave us the coefficient matrix. The official Mathematics for Machine Learning companion places this lesson in Section 2.3 of the current book PDF. The numbers, feature scenario, code, visual, and assessments here are original.

A staircase records what the equations can determine

Start with a matrix $Ainmathbb{R}^{m imes n}$ . A matrix is in row-echelon form when three structural rules hold:

Every nonzero row appears above every all-zero row.
The first nonzero entry of each nonzero row, called its pivot, appears farther right than the pivot in the row above it.
Every entry below a pivot is zero.

The pivots form a left-to-right staircase. A pivot column is a direction that the rows constrain. A column without a pivot is a free direction: its value can vary while the equations determine the other variables around it.

Row-echelon form is not the same as reduced row-echelon form. Reduced form also requires each pivot to equal $1$ and to be the only nonzero entry in its column. Either form can reveal rank and free variables; reduced form is often more convenient for reading a parameterized answer.

The visual below keeps the intermediate states visible. The final matrix has two pivots, in columns $f_1$ and $f_2$ ; column $f_3$ has no pivot.

Static row-reduction record

The pivots form a staircase

Each state keeps the complete matrix visible. The final nonzero row has its pivot farther right than the row above it, and the zero row has moved to the bottom.

Step 0Original feature matrixStart
Original feature matrix
Row f₁ f₂ f₃
R1 1 2 3
R2 2 4 6
R3 1 1 2
Step 1Clear the first column below the pivotR₂ ← R₂ − 2R₁; R₃ ← R₃ − R₁
Clear the first column below the pivot
Row f₁ f₂ f₃
R1 1 2 3
R2 0 0 0
R3 0 -1 -1
Step 2Move the nonzero row upwardR₂ ↔ R₃
Move the nonzero row upward
Row f₁ f₂ f₃
R1 1 2 3
R2 0 -1 -1
R3 0 0 0
Step 3Normalize the second pivotR₂ ← −R₂
Normalize the second pivot
Row f₁ f₂ f₃
R1 1 2 3
R2 0 1 1
R3 0 0 0

Original feature matrix
Row	f₁	f₂	f₃
R1	1	2	3
R2	2	4	6
R3	1	1	2

Clear the first column below the pivot
Row	f₁	f₂	f₃
R1	1	2	3
R2	0	0	0
R3	0	-1	-1

Move the nonzero row upward
Row	f₁	f₂	f₃
R1	1	2	3
R2	0	-1	-1
R3	0	0	0

Normalize the second pivot
Row	f₁	f₂	f₃
R1	1	2	3
R2	0	1	1
R3	0	0	0

The highlighted entries are pivots in the final structure. Column f₃ has no pivot, so it is the free direction that exposes the redundant feature.

Row operations expose two independent feature directions and one redundant column. The underlined entries mark the final pivot positions.

Worked example: a redundant feature column

Imagine a tiny design matrix whose columns summarize three signals extracted for three examples:

$f_1$ : retrieval-match strength;
$f_2$ : verification strength; and
$f_3$ : a combined signal that was accidentally logged as a separate feature.

The matrix is

A= \begin{bmatrix} 1&2&3\\ 2&4&6\\ 1&1&2 \end{bmatrix}.

Before doing any reduction, notice that the third column is the sum of the first two: $f_3=f_1+f_2$ . That is a useful intuition, but row reduction gives us a procedure that still works when the relationship is not obvious by inspection.

First use the top-left entry as a pivot. Clear the entries beneath it:

R_2\leftarrow R_2-2R_1, \qquad R_3\leftarrow R_3-R_1,

which gives

\begin{bmatrix} 1&2&3\\ 0&0&0\\ 0&-1&-1 \end{bmatrix}.

The second row is temporarily zero, but the system has not lost information: the nonzero row below still needs to be moved upward. Swap the rows and then multiply the new second row by $-1$ :

\begin{bmatrix} 1&2&3\\ 0&-1&-1\\ 0&0&0 \end{bmatrix} \quad\longrightarrow\quad \boxed{ \begin{bmatrix} 1&2&3\\ 0&1&1\\ 0&0&0 \end{bmatrix}}.

The pivots are in columns 1 and 2. Column 3 has no pivot, so it is the free column. The matrix has rank 2 rather than 3: the data contain three named features but only two independent directions.

Free variables describe a family, not a failure

To see what the missing pivot means, attach a right-hand side that is consistent with the same row relationships:

\begin{bmatrix} 1&2&3&\mid&4\\ 2&4&6&\mid&8\\ 1&1&2&\mid&3 \end{bmatrix}.

The same operations produce

\begin{bmatrix} 1&0&1&\mid&2\\ 0&1&1&\mid&1\\ 0&0&0&\mid&0 \end{bmatrix}.

The equations are now

\theta_1+\theta_3=2, \qquad \theta_2+\theta_3=1.

Let the free variable be $\theta_3=t$ . Then

\theta_1=2-t, \qquad \theta_2=1-t, \qquad \theta_3=t.

Every value of $t$ produces a parameter vector that gives the same right-hand side. For example, $t=0$ gives $(2,1,0)$ , while $t=1$ gives $(1,0,1)$ . Both satisfy the three original equations. The model’s predictions are fixed, but the individual weights are not identifiable from these observations alone.

A row of zeros is different from a contradiction

The last row in the example becomes

0\theta_1+0\theta_2+0\theta_3=0.

That row is redundant, so it contributes no new constraint. Do not confuse it with

0\theta_1+0\theta_2+0\theta_3=5,

which is impossible. In an augmented matrix, a row of zeros on the left with a nonzero right-hand side signals inconsistency and therefore no solution. The same left-hand matrix can support infinitely many solutions, one solution, or no solution depending on the transformed right-hand side.

This distinction matters in data pipelines. Repeated or dependent rows may reduce the amount of independent evidence without making the equations contradictory. A contradictory row instead says that the observations cannot be explained by the proposed linear model exactly.

Implement the pivot search in NumPy

The following compact implementation uses partial pivoting. At each column it chooses the largest-magnitude candidate below the current pivot row, swaps it into place, normalizes the pivot, and clears entries below it. The tolerance prevents a value that is only round-off noise from being treated as a real pivot.

import numpy as np


def row_echelon(matrix, tolerance=1e-12):
    a = np.array(matrix, dtype=float, copy=True)
    rows, columns = a.shape
    pivot_row = 0
    pivot_columns = []

    for column in range(columns):
        if pivot_row == rows:
            break

        candidate = pivot_row + np.argmax(np.abs(a[pivot_row:, column]))
        if abs(a[candidate, column]) <= tolerance:
            continue

        a[[pivot_row, candidate]] = a[[candidate, pivot_row]]
        a[pivot_row] /= a[pivot_row, column]

        for row in range(pivot_row + 1, rows):
            a[row] -= a[row, column] * a[pivot_row]

        a[np.abs(a) <= tolerance] = 0
        pivot_columns.append(column)
        pivot_row += 1

    return a, pivot_columns


features = np.array([
    [1, 2, 3],
    [2, 4, 6],
    [1, 1, 2],
])

echelon, pivots = row_echelon(features)
print(echelon)
print(pivots)
print(np.linalg.matrix_rank(features))

The meaningful output is approximately

[[1. 2. 3.]
 [0. 1. 1.]
 [0. 0. 0.]]
[0, 1]
2

The exact decimals depend on the dtype and tolerance, but the structural conclusion is stable here: two pivot columns and one free column. NumPy’s matrix_rank also uses a tolerance around singular values rather than asking whether a floating-point value is literally equal to zero. That is why a rank decision should be reported with its scale and tolerance, not as an eternal truth about noisy measurements.

What this means for an AI model

Suppose a linear model computes

\hat{y}=\theta_1f_1+\theta_2f_2+\theta_3f_3.

When $f_3=f_1+f_2$ , the prediction becomes

\hat{y}=(\theta_1+\theta_3)f_1+(\theta_2+\theta_3)f_2.

Only the sums $\theta_1+\theta_3$ and $\theta_2+\theta_3$ are observable in this design. Different parameter vectors can therefore produce identical predictions on every row of the matrix. A training run may still return one set of weights because an optimizer, initialization, regularizer, or numerical path selects one point from the family. That selected point is not proof that the data identified each feature’s independent contribution.

Use row-echelon form as a diagnostic, not as a replacement for validation. A rank calculation on a small sample can change when preprocessing, scaling, missing-value handling, or new data changes. For a larger or noisy matrix, inspect the singular-value scale and condition number as well. A full-rank matrix can still be nearly dependent, making its solution sensitive to small measurement changes.

condition = np.linalg.cond(features)
residual = np.linalg.norm(features[:, 2] - features[:, 0] - features[:, 1])
print(condition, residual)

Here the residual is exactly zero in the stored values and the condition number is extremely large in floating-point arithmetic (about $3.8\times10^{16}$ in this run). An exactly rank-deficient matrix has an infinite mathematical condition number, but a floating-point SVD may report a very large finite value. That contrast is itself a warning. In production data, use a scale-aware tolerance: a tiny residual can mean a genuine near-dependency or just a measurement below the precision that matters for the application.