Math for AI course · Lesson 52 of 180

Least Squares Is a Projection Problem

Derive a least-squares line fit as an orthogonal projection, verify residuals in NumPy, and see why solving normal equations beats forming an explicit inverse.

Share this article

Suppose a model is allowed to draw only a straight line through a set of observations. Which line gives the closest predictions, and what does “closest” mean when the targets are stored as a vector rather than as points on a graph?

Ordinary least squares answers both questions with geometry. It chooses the prediction vector inside the design matrix’s column space that is nearest to the target vector. The difference between those two vectors is the residual, and the residual is perpendicular to every feature direction used by the fit.

By the end of this lesson, you will be able to write a line fit as a matrix problem, derive the normal equations from residual orthogonality, calculate a small fit by hand, verify it with NumPy, and recognize when the calculation is limited by rank or by the model’s choice of features.

Learning outcome: Derive normal-equation geometry without relying on explicit inversion.

This lesson connects Lesson 025: Numerical Solving Without Explicit Inverses, where a linear system became a problem for a solver rather than a matrix inverse, with Lesson 050: Projection Finds the Nearest Point and Lesson 051: Projection Matrices Transform Whole Datasets. Section 3.8 of the official Mathematics for Machine Learning book provides the curricular grounding. The data, arithmetic, visual, NumPy lab, AI connection, and assessments here are original.

A line fit becomes a subspace of prediction vectors

Start with mm observations. Observation ii has one input feature xix_i and a numeric target yiy_i. A line with an intercept predicts

y^i=β0+β1xi,\hat y_i=\beta_0+\beta_1x_i,

where β0\beta_0 is the intercept and β1\beta_1 is the slope. The hat on y^i\hat y_i means “prediction,” not a new measured target.

To expose the geometry, put the observations into a design matrix:

X=[1x11x2⋮⋮1xm]∈Rm×2.X= \begin{bmatrix} 1 & x_1\\ 1 & x_2\\ \vdots & \vdots\\ 1 & x_m \end{bmatrix} \in\mathbb{R}^{m\times 2}. β=[β0β1]∈R2.\beta=\begin{bmatrix}\beta_0\\\beta_1\end{bmatrix} \in\mathbb{R}^{2}. y=[y1y2⋮ym]∈Rm.y=\begin{bmatrix}y_1\\y_2\\\vdots\\y_m\end{bmatrix} \in\mathbb{R}^{m}.

The first column of XX is a column of ones. It gives the model a free offset; without it, the line would be forced through the origin. Matrix multiplication produces the whole prediction vector in one operation:

y^=Xβ=β0[11⋮1]+β1[x1x2⋮xm].\hat y=X\beta=\beta_0 \begin{bmatrix}1\\1\\\vdots\\1\end{bmatrix} +\beta_1 \begin{bmatrix}x_1\\x_2\\\vdots\\x_m\end{bmatrix}.

So every possible line prediction lies in the span of the two columns of XX. That span is a subspace of Rm\mathbb{R}^{m}, the observation space. The coordinates of a point in this space correspond to the observations, not to the two model parameters.

This is a statement about finite-dimensional Euclidean geometry. It does not assume that the residuals are Gaussian, independent, or even “noise.” Those assumptions can matter in a statistical analysis, but the projection result comes from the chosen squared-error distance and the feature columns.

Perpendicular residuals produce the normal equations

If y^\hat y is the closest point in col⁡(X)\operatorname{col}(X), then the residual r=y−y^r=y-\hat y must be perpendicular to every direction in that subspace. In particular, it must be perpendicular to each column of XX. Writing all of those dot products together gives

XTr=0.X^{\mathsf T}r=0.

Substitute r=y−Xβ^r=y-X\hat\beta:

XT(y−Xβ^)=0XTy−XTXβ^=0XTXβ^=XTy.\begin{aligned} X^{\mathsf T}(y-X\hat\beta)&=0\\ X^{\mathsf T}y-X^{\mathsf T}X\hat\beta&=0\\ \boxed{X^{\mathsf T}X\hat\beta=X^{\mathsf T}y}.& \end{aligned}

These are the normal equations. The left side is a 2×22\times2 matrix times a two-entry coefficient vector, and the right side is a two-entry vector. They are not a statistical spell; they are the algebraic form of the right-angle condition.

Why does perpendicularity give the nearest point rather than merely a point that happens to pass a dot-product test? Take any other coefficient vector β\beta and compare its prediction with y^\hat y:

y−Xβ=(y−Xβ^)⏟r+X(β^−β)⏟a direction in col⁡(X).y-X\beta =\underbrace{(y-X\hat\beta)}_{r} +\underbrace{X(\hat\beta-\beta)}_{\text{a direction in }\operatorname{col}(X)}.

The first term is perpendicular to the second because XTr=0X^{\mathsf T}r=0. The Pythagorean theorem therefore gives

∥y−Xβ∥22=∥r∥22+∥X(β^−β)∥22≥∥r∥22.\lVert y-X\beta\rVert_2^2 =\lVert r\rVert_2^2 +\lVert X(\hat\beta-\beta)\rVert_2^2 \geq\lVert r\rVert_2^2.

The fitted prediction cannot be beaten by another vector in the same column space. If the two columns of XX are linearly independent, the coefficients are unique too. If they are dependent, the projected prediction can still be well-defined even though multiple coefficient vectors describe it.

A four-observation line fit, calculated by hand

Use four observations with a single feature:

x=[0123],y=[1224].x=\begin{bmatrix}0\\1\\2\\3\end{bmatrix}, \qquad y=\begin{bmatrix}1\\2\\2\\4\end{bmatrix}.

The points do not fall exactly on one line, which makes the residual geometry visible. With the intercept column included,

X=[10111213],X∈R4×2.X= \begin{bmatrix} 1&0\\ 1&1\\ 1&2\\ 1&3 \end{bmatrix}, \qquad X\in\mathbb{R}^{4\times2}.

The first orthogonality check has a useful interpretation: because this model contains an intercept, the residuals sum to zero. The second says the residual has no remaining linear alignment with the feature values. Neither statement says that each residual is small or that the line will generalize to new inputs.

A target vector decomposed by a line fitThe component view compares each observed target with its fitted value. The observation-space schematic shows the target as the sum of a prediction in the design subspace and an orthogonal residual.

Coordinate-by-coordinate view

Observed targets, fitted predictions, and residual gaps Four target values are connected by a blue line and four fitted values by a teal line. Amber dashed segments connect each prediction to its target and show the residual for that observation.-101234i = 0i = 1i = 2i = 3observation index ivalue
  • target y
  • prediction ŷ
  • residual r = y − ŷ

Observation-space decomposition

Target equals fitted prediction plus orthogonal residual A schematic vector diagram shows the fitted prediction arrow along the horizontal design-subspace direction, the residual arrow rising vertically from its tip, and the target arrow from the origin to the final point. A right-angle mark identifies the residual as perpendicular to the fitted subspace.ŷryright angle
The four coordinates used in the decomposition
ixᵢyᵢŷᵢrᵢ
001.00.9+0.1
112.01.8+0.2
222.02.7-0.7
334.03.6+0.4

Orthogonality: Xᵀr = [0, 0]

Squared residual length: ‖r‖₂² = 0.70

The right panel uses schematic orthogonal coordinates for vectors in ℝ⁴. The table keeps the exact four components visible, so the visual does not ask a two-dimensional drawing to stand in for the full observation space.

The same projection in executable NumPy

The hand calculation translates directly into a small NumPy program. With one sample per row, XX has shape (m,2)(m,2) and β\beta has shape (2,)(2,), so X @ beta has shape $(m,). NumPy's [matmul` documentation](https://numpy.org/doc/stable/reference/generated/numpy.matmul.html) describes this shape-aware matrix product.

import numpy as np

x = np.array([0.0, 1.0, 2.0, 3.0])
y = np.array([1.0, 2.0, 2.0, 4.0])

# A column of ones gives the line an intercept.
X = np.column_stack((np.ones_like(x), x))
normal_matrix = X.T @ X
normal_rhs = X.T @ y

# Solve (X.T @ X) @ beta = X.T @ y without forming an inverse.
beta = np.linalg.solve(normal_matrix, normal_rhs)
y_hat = X @ beta
residual = y - y_hat

print("X.T @ X =\n", normal_matrix)
print("X.T @ y =", normal_rhs)
print("beta =", beta)
print("y_hat =", y_hat)
print("residual =", residual)
print("X.T @ residual =", X.T @ residual)
print("squared residual length =", np.linalg.norm(residual) ** 2)

assert np.allclose(X.T @ residual, np.zeros(X.shape[1]))

The output is:

X.T @ X =
 [[ 4.  6.]
  [ 6. 14.]]
X.T @ y = [ 9. 18.]
beta = [0.9 0.9]
y_hat = [0.9 1.8 2.7 3.6]
residual = [ 0.1  0.2 -0.7  0.4]
X.T @ residual = [0. 0.]
squared residual length = 0.7

numpy.linalg.solve solves a square linear system for the coefficient vector. In this example, XTXX^{\mathsf T}X is square and nonsingular, so it is a suitable teaching example. The assert uses numpy.allclose because floating-point arithmetic can leave a tiny value where the exact mathematics says zero. The error size uses numpy.linalg.norm, which computes the Euclidean length when used with its default vector norm.

Why not multiply by an explicit inverse?

For this small example, the following diagnostic produces the same coefficient vector up to floating-point tolerance:

beta_via_inverse = np.linalg.inv(normal_matrix) @ normal_rhs
print(np.allclose(beta, beta_via_inverse))  # True for this toy system

That agreement does not make the inverse the preferred implementation. An inverse performs more work than solving for the one right-hand side, and forming it can expose the calculation to avoidable numerical error. This is the same implementation lesson as Lesson 025: express the system you need and use a solver designed for that system.

What the geometry says about an AI model

In a small regression baseline, the columns of XX are the feature directions the model is allowed to combine. The target vector yy contains one desired numeric outcome per training example. Least squares finds the closest outcome vector that those feature directions can express. The residual is the part of the training target that this feature set leaves outside its span.

That perspective makes two evaluation habits concrete:

  • A small training residual means the chosen feature space can reconstruct the observed targets closely under squared error. It does not by itself measure performance on new examples.
  • A large residual can mean noisy outcomes, a missing feature, an outlier, or a line that is too restrictive for the relationship. The geometry identifies what the current model cannot express; it does not identify the cause.

The intercept also encodes a modeling choice. With a column of ones, the fit can shift vertically and the residuals sum to zero. Removing that column forces the prediction vector into the span of xx alone, which may be appropriate for a physical law through the origin but is an additional constraint for most datasets.

Boundaries that change the calculation

Orthogonal does not mean zero. The residual is perpendicular to the feature columns, not necessarily absent. In the worked example, the residual at x=2x=2 is −0.7-0.7, even though the full residual vector is orthogonal to both columns. A line can be the closest available model while missing every observation by a nonzero amount.

Rank deficiency changes coefficient uniqueness. If the second column of XX duplicated the intercept column—for example, if every xix_i were the same— then the columns would not provide two independent directions. The matrix XTXX^{\mathsf T}X would be singular, and np.linalg.solve would reject the normal system. The prediction subspace would still exist, but the split between intercept and slope would not be unique. Later lessons will study other solvers; adding an arbitrary tiny number to the diagonal would define a different regularized problem rather than repair the original geometry.

Ill-conditioned is not the same as singular. Nearly duplicate columns can make XTXX^{\mathsf T}X extremely sensitive to small changes in the data. A solver can return coefficients while those coefficients remain unstable. This is a warning to inspect feature relationships and residual behavior, not a reason to trust a plausible-looking number blindly.

The intercept is not automatic. If the model intentionally has no intercept, then the normal equations still enforce orthogonality to the available feature columns, but there is no all-ones column and therefore no reason for the residuals to sum to zero. Changing the columns changes the projection space and the question being answered.

Least squares is therefore more specific than “draw the best line.” It is a projection onto the set of predictions the design matrix can express, using Euclidean squared error as the distance. The next regression lessons can build on that geometry, but they should not quietly turn it into a claim that every target relationship is linear or that every residual is random noise.

Check your understanding

Question 1

For m observations and one input feature with an intercept, what are the shapes of X, β, y, and Xβ?

Show the step-by-step solution

The design matrix has one row per observation and two columns: the all-ones intercept column and the feature column. Therefore X∈Rm×2X\in\mathbb{R}^{m\times2}. The coefficient vector contains β0\beta_0 and β1\beta_1, so β∈R2\beta\in\mathbb{R}^{2}. The target vector has one entry per observation, so y∈Rmy\in\mathbb{R}^{m}. Matrix multiplication then gives

(m×2)(2×1)=m×1,(m\times2)(2\times1)=m\times1,

so XβX\beta is an mm-entry prediction vector. The two parameters describe the coordinates used to construct a point in the mm-dimensional observation space.

Question 2

For X = [[1, 0], [1, 1], [1, 2]] and y = [1, 2, 2], compute XᵀX and Xᵀy.

Show the step-by-step solution

Compute the column inner products. The first column is [1,1,1]T[1,1,1]^{\mathsf T} and the second is [0,1,2]T[0,1,2]^{\mathsf T}:

XTX=[12+12+121⋅0+1⋅1+1⋅20⋅1+1⋅1+2⋅102+12+22]=[3335].X^{\mathsf T}X =\begin{bmatrix} 1^2+1^2+1^2 & 1\cdot0+1\cdot1+1\cdot2\\ 0\cdot1+1\cdot1+2\cdot1 & 0^2+1^2+2^2 \end{bmatrix} =\begin{bmatrix}3&3\\3&5\end{bmatrix}.

Now correlate each column with yy:

XTy=[1+2+20⋅1+1⋅2+2⋅2]=[56].X^{\mathsf T}y =\begin{bmatrix} 1+2+2\\0\cdot1+1\cdot2+2\cdot2 \end{bmatrix} =\begin{bmatrix}5\\6\end{bmatrix}.

These are the matrix and right-hand side used by the normal equations.

Question 3

Using the worked lesson example, verify by hand that r = [0.1, 0.2, −0.7, 0.4] is orthogonal to both columns of X.

Show the step-by-step solution

The first column is the all-ones vector. Its dot product with rr is

1(0.1)+1(0.2)+1(−0.7)+1(0.4)=0.1(0.1)+1(0.2)+1(-0.7)+1(0.4)=0.

The second column is [0,1,2,3]T[0,1,2,3]^{\mathsf T}. Its dot product is

0(0.1)+1(0.2)+2(−0.7)+3(0.4)=0.2−1.4+1.2=0.0(0.1)+1(0.2)+2(-0.7)+3(0.4)=0.2-1.4+1.2=0.

Both dot products are zero, so stacking them gives XTr=[0,0]TX^{\mathsf T}r=[0,0]^{\mathsf T}. The residual is orthogonal to every linear combination of the two columns as well.

Question 4

Why does Xᵀr = 0 imply that the fitted prediction is at least as close to y as any other prediction Xβ?

Show the step-by-step solution

Let β^\hat\beta be the fitted coefficients and let β\beta be any alternative. Rewrite the alternative residual as

y−Xβ=(y−Xβ^)+X(β^−β)=r+X(β^−β).y-X\beta=(y-X\hat\beta)+X(\hat\beta-\beta)=r+X(\hat\beta-\beta).

The first term is rr. The second is in col⁡(X)\operatorname{col}(X). Since XTr=0X^{\mathsf T}r=0, rr is perpendicular to that second term. The squared length therefore splits by Pythagoras:

∥y−Xβ∥22=∥r∥22+∥X(β^−β)∥22≥∥r∥22.\lVert y-X\beta\rVert_2^2 =\lVert r\rVert_2^2+\lVert X(\hat\beta-\beta)\rVert_2^2 \geq\lVert r\rVert_2^2.

The extra term cannot be negative, so no other prediction in the column space has a smaller squared distance to yy.

Question 5

An engineer says, ‘A least-squares fit has residuals that are all zero because the residual is orthogonal.’ What is wrong with the claim?

Show the step-by-step solution

Orthogonality means a dot product is zero; it does not mean the residual vector itself is zero. In the lesson example,

r=[0.1,0.2,−0.7,0.4]r=[0.1,0.2,-0.7,0.4]

has nonzero entries, but its dot products with the all-ones column and the feature column both vanish. The zero vector would occur only when yy already lies in col⁡(X)\operatorname{col}(X), meaning the available line model can reproduce the targets exactly.

Question 6

What changes if the intercept column of ones is removed from X?

Show the step-by-step solution

Without the ones column, the model becomes y^i=β1xi\hat y_i=\beta_1x_i, a line forced through the origin. The column space becomes span⁡{x}\operatorname{span}\{x\} rather than the span of both the constant vector and xx. The normal equation still requires the residual to be orthogonal to the remaining feature column, but there is no all-ones column to enforce

1Tr=0.\mathbf{1}^{\mathsf T}r=0.

Therefore the residuals no longer have to sum to zero. This is a modeling constraint, not a harmless formatting change.

Question 7

Why can np.linalg.solve fail when two columns of X are duplicates, even though a closest prediction still exists?

Show the step-by-step solution

Duplicate columns do not create two independent feature directions. If X=[1 1]X=[\mathbf{1}\ \mathbf{1}], then the columns span only one direction, and XTXX^{\mathsf T}X is singular. A singular matrix has no unique coefficient solution for the normal equations, so np.linalg.solve correctly refuses the system instead of inventing a unique split between the two coefficients.

The projection of yy onto the one-dimensional column space can still be unique. What is ambiguous is the coordinate representation: many pairs (β0,β1)(\beta_0,\beta_1) can add to the same constant prediction. A different algorithm or an explicitly regularized problem is needed to choose among such representations.

Question 8

A model has a tiny training residual after fitting many features. Why is that not enough evidence that it will predict new AI examples well?

Show the step-by-step solution

Least squares only finds the closest target vector inside the span of the training design matrix. Adding features can make that span expressive enough to pass close to the observed targets, which reduces training error. New examples may have different feature values or target behavior, and their targets were not part of the projection that was computed.

To assess generalization, keep a separate evaluation set or use an appropriate cross-validation design, then measure predictions on examples not used to fit β^\hat\beta. The training residual remains useful for diagnosing the fit, but it is not a guarantee about future observations.

The practical habit is simple: write down the prediction space before trusting the coefficient vector. Least squares gives the nearest point in that space; the residual tells you exactly which part of the observed target remains outside the model’s chosen span.

Sources

  1. Mathematics for Machine Learning official PDF, Section 3.8
  2. NumPy documentation: numpy.linalg.solve
  3. NumPy documentation: numpy.matmul
  4. NumPy documentation: numpy.allclose
  5. NumPy documentation: numpy.linalg.norm