Matrix Rank Counts Independent Information in AI Features

A dataset can have three feature columns and still contain only two independent directions of variation. That difference matters when a model tries to assign separate effects to features that always move together. Matrix rank counts the independent directions in a matrix, whether we inspect its feature columns or its sample rows.

By the end, you will be able to compute a matrix’s rank by finding pivots, explain why row and column perspectives give the same number, and read a singular-value preview without treating every small value as exactly zero.

Learning outcome: Compute rank and connect row and column perspectives.

This lesson brings together the row-echelon form and dimension ideas: pivots identify independent directions, and rank counts them. Section 2.6.2 of Mathematics for Machine Learning defines rank through independent rows and columns. The data matrix, feature interpretation, singular-value chart, calculations, and exercises here are original.

Three columns can carry only two directions

Imagine four sessions described by minutes on a mobile app, minutes on a website, and total active minutes. Store each session as a row and each measurement as a column:

X=\begin{bmatrix} 1&2&3\\ 2&4&6\\ 0&1&1\\ 1&3&4 \end{bmatrix} = \begin{bmatrix} \vert&\vert&\vert\\ \boldsymbol{x}_{\text{mobile}}& \boldsymbol{x}_{\text{web}}& \boldsymbol{x}_{\text{total}}\\ \vert&\vert&\vert \end{bmatrix} \in\mathbb{R}^{4\times3}.

The third feature is calculated as mobile time plus website time in every row:

\boldsymbol{x}_{\text{total}} =\boldsymbol{x}_{\text{mobile}}+\boldsymbol{x}_{\text{web}}.

So the three columns are dependent. The total column is useful as a named measurement, but it adds no new linear direction to this particular matrix. The first two columns are independent: in row three, only the website column is nonzero; then row one rules out a nonzero multiple of the mobile column. Thus the column space has dimension two.

Here $m=4$ sessions and $n=3$ features, so rank can be at most three. We have already shown that the three columns depend on one another while the first two do not, giving $\operatorname{rank}(X)=2$ .

Pivots calculate rank from rows

For a linear system, a row operation must also be applied to its right-hand side to preserve the solutions. On this feature matrix, elimination preserves rank and reveals the pivots. First replace row 2 with row 2 minus twice row 1, and row 4 with row 4 minus row 1. Then subtract row 3 from the new row 4 and move the nonzero third row into the second position:

\begin{bmatrix} 1&2&3\\ 2&4&6\\ 0&1&1\\ 1&3&4 \end{bmatrix} \sim \begin{bmatrix} 1&2&3\\ 0&1&1\\ 0&0&0\\ 0&0&0 \end{bmatrix} \sim \begin{bmatrix} 1&0&1\\ 0&1&1\\ 0&0&0\\ 0&0&0 \end{bmatrix}.

There are two nonzero rows and two pivot columns, so the rank is two. Columns one and two of the original matrix form a basis for its column space; the pivot in column three would be absent because that column is the sum of the first two.

The row view reaches the same count directly. Write the sample rows as $\boldsymbol{r}_1,\ldots,\boldsymbol{r}_4$ . The second row equals $2\boldsymbol{r}_1$ , and the fourth equals $\boldsymbol{r}_1+\boldsymbol{r}_3$ . Rows one and three are independent, so the row space also has dimension two. In general, this equality of row rank and column rank holds for every finite matrix, including rectangular ones; transposing swaps the two spaces but cannot change how many independent directions they contain.

Singular values preview the directions

For noisy or large matrices, software commonly estimates rank from a singular value decomposition (SVD). The SVD represents a matrix as $X=U\Sigma V^\mathsf{T}$ , where the diagonal entries of $\Sigma$ are nonnegative singular values sorted from largest to smallest. Each value tells how strongly the matrix acts along a paired direction. An exact zero means that direction is absent.

For this example, the singular values are approximately $9.882$ , $0.581$ , and zero (up to floating-point roundoff). Dividing by the largest value makes a scale-free preview: $1.000$ , $0.059$ , and approximately $0$ . The third bar vanishes because the total feature is an exact sum. The second direction is smaller than the first, but it is still nonzero. These ratios are not percentages of variance explained: singular values show scale, while their squares are eigenvalues of $X^\mathsf{T}X$ . For centered data, those squares track variance along the corresponding directions, up to a shared normalization.

Singular-value preview

Two directions remain in the feature matrix

Each bar shows a singular value divided by the largest one. The third direction is zero up to floating-point roundoff because total time is the sum of the other features.

First direction1.000
Second direction0.059
Third direction≈ 0

The values are approximately 9.882, 0.581, and 0 before normalization. A small direction is not automatically a zero direction; rank needs an exact or numerical rule.

The chart is a preview of one matrix, not a universal cutoff. NumPy’s matrix_rank counts singular values above a tolerance. By default, that tolerance scales with the largest singular value, the matrix’s largest dimension, and the floating-point precision of the data type. This handles roundoff in exact dependencies; a different tolerance may be more suitable when measurement noise is larger than floating-point error. The svd routine returns singular values in descending order.

import numpy as np

X = np.array([
    [1, 2, 3],
    [2, 4, 6],
    [0, 1, 1],
    [1, 3, 4],
], dtype=float)

singular_values = np.linalg.svd(X, compute_uv=False)
rank_columns = np.linalg.matrix_rank(X)
rank_rows = np.linalg.matrix_rank(X.T)

print(X.shape)                              # (4, 3)
print(np.round(singular_values, 3))          # [9.882 0.581 0.   ]
print(rank_columns, rank_rows)               # 2 2

Floating-point SVD may return a tiny positive number where exact arithmetic gives zero. NumPy’s tolerance classifies that value as zero for the rank estimate. The displayed 0. is rounded output, not a claim that every computer’s computed singular value is bit-for-bit zero.

Rank identifies redundant model weights

An intercept-free linear model with weights $\boldsymbol{w}\in\mathbb{R}^3$ predicts the four session outcomes as $\hat{\boldsymbol{y}}=X\boldsymbol{w}$ . Let the weights be $(w_m,w_w,w_t)$ for mobile, web, and total time. Since total time is the sum of the first two features,

X\boldsymbol{w} = (w_m+w_t)\boldsymbol{x}_{\text{mobile}} + (w_w+w_t)\boldsymbol{x}_{\text{web}}.

Only two combined weights affect predictions, even though the model stores three. For any number $c$ , replacing $(w_m,w_w,w_t)$ with $(w_m+c,w_w+c,w_t-c)$ leaves every prediction unchanged. On this dataset, rank two therefore means that the three coefficients cannot be uniquely identified from the predictions alone. Regularization can select one set of weights, but the rank calculation by itself does not say which feature deserves a causal or semantic interpretation.

That is the AI application: rank measures effective linear feature diversity in a data matrix. It can expose redundant derived features before they complicate coefficient interpretation or least-squares fitting. The number does not measure all information in a dataset, predict model accuracy, or capture nonlinear relationships. Changing the data rows, feature definitions, or preprocessing can change the rank.

Near dependence needs a stated tolerance

The constructed total is an exact sum. Real measurements may instead satisfy $\boldsymbol{x}_3\approx\boldsymbol{x}_1+\boldsymbol{x}_2$ . An exact mathematical rank could then be three, even when the smallest singular value is tiny. A numerical rank may count that direction as negligible when it falls below a chosen tolerance. Report the tolerance and the data scale when that distinction affects a decision.

Feature scaling also changes singular-value sizes. Multiplying a feature column by a nonzero constant preserves exact rank, but it can alter the singular-value ratios used for a numerical or approximate-rank judgment. Standardization is therefore not cosmetic when interpreting small singular values. Rank describes linear dependence in the represented matrix under the chosen encoding; it does not award a fixed amount of semantic information to each column.

When a matrix has $p$ feature columns but only $r$ independent directions, the row-echelon form shows which columns can serve as a basis, while the singular values show how strong those directions appear numerically. The next step is to ask what a matrix maps to and which input directions it loses: the kernel and image make those two questions explicit.

Check your understanding

Question 1

For a 5 × 3 matrix, what is the largest possible rank, and what does a rank of 2 imply about its columns and rows?