Orthogonality Means No Linear Overlap

Suppose an AI representation contains two directions that should add different things to a later calculation. If one direction contributes nothing along the other, their inner product is zero. For example, with $u=(3,1)$ and $v=(-1,3)$,

$$u^{\mathsf T}v=3(-1)+1(3)=-3+3=0.$$

Neither vector is empty, but neither has a component along the other under the usual dot-product geometry. That is the useful idea behind orthogonality: zero inner product means no linear overlap in the chosen geometry.

By the end of this lesson, you will be able to test orthogonality by hand and in NumPy, interpret the zero as a right angle and a zero projection, and keep orthogonality separate from correlation, independence, and “no information.”

Learning outcome: Test orthogonality and interpret zero inner product geometrically.

This lesson builds on Lesson 041: Inner Products Measure Alignment, which introduced the dot product as an alignment score. Lesson 044: Angles and Cosine Similarity then separated direction from vector length. Section 3.4 of the official Mathematics for Machine Learning book is the mathematical reference for this lesson. The explanations, numbers, code, visual, AI connection, and assessments are original.

Orthogonality is a zero inner product

Let $x=(x_1,\ldots,x_n)$ and $y=(y_1,\ldots,y_n)$ be real vectors with the same number of coordinates. The coordinate at position $i$ must describe the same kind of quantity in both vectors. In this lesson the inner product is the standard Euclidean dot product.

This is an algebraic definition. In the ordinary Euclidean plane, it matches the familiar picture of a right angle for nonzero vectors. The angle formula from the previous lesson makes the connection explicit:

$$\cos(\theta)= \frac{x^{\mathsf T}y}{\lVert x\rVert_2\lVert y\rVert_2}.$$

If both vectors are nonzero and $x^{\mathsf T}y=0$, the numerator is zero and the denominator is positive. Therefore $\cos(\theta)=0$ and $\theta=\pi/2=90^\circ$. The nonzero condition matters: a zero vector has zero dot product with everything, but it has no direction and therefore no angle to interpret.

A worked test from coordinates

Return to $u=(3,1)$ and $v=(-1,3)$. The two entries are coordinates, not probabilities or degrees. Work through the calculation in three passes.

The cancellation is the central event. The positive product from the second coordinate exactly cancels the negative product from the first. In a linear score that uses $u$ as a direction, $v$ contributes no amount along $u$.

One dot product, three geometric relationships

The table is the complete numerical fallback for the right-angle visual.

First vector	Second vector	Dot product	Angle	Interpretation
u = (3, 1)	v = (-1, 3)	0.0	90.0°	dot = 0; angle = 90°
u = (3, 1)	w = (6, 2)	20.0	0.0°	dot = 20; angle = 0°
u = (3, 1)	z = (2, 3)	9.0	37.9°	dot = 9; angle ≈ 37.9°

Perpendicular

Same direction

Oblique

The table carries every number. Solid versus dashed strokes, vector labels, the right-angle square, and the written relationship give the diagrams redundant cues beyond color.

The visual keeps three ideas side by side. The aligned pair $w=(6,2)$ is a positive scaling of $u$, so it has angle $0^\circ$ and dot product $3(6)+1(2)=20$. The oblique pair $z=(2,3)$ has dot product $3(2)+1(3)=9$ and cosine

$$\frac{9}{\sqrt{10}\sqrt{13}}\approx0.789,$$

so its angle is about $37.9^\circ$. A zero dot product is therefore a special boundary between positive and negative directional alignment; it is not just a small positive score.

Scaling changes size, not a zero relationship

For scalars $a$ and $b$, bilinearity gives

$$(ax)^{\mathsf T}(by)=ab\,x^{\mathsf T}y.$$

If $x^{\mathsf T}y=0$, then the scaled vectors remain orthogonal for any finite $a$ and $b$. For example,

$$(-4u)^{\mathsf T}(7v)=(-28)(u^{\mathsf T}v)=(-28)(0)=0.$$

The lengths and the arrow directions can change: a negative scalar reverses an arrow, and a larger absolute scalar makes it longer. For nonzero scalars, a right angle stays a right angle when one or both sides are stretched or reversed. Orthogonality is about the zero of the inner product, not about the vectors having equal length.

A residual can be orthogonal without being empty

Orthogonality becomes especially useful when a calculation removes one linear direction from a vector. Let $u\ne0$ be a direction and let $y$ be an observation. The projection of $y$ onto the line spanned by $u$ is

$$\operatorname{proj}_{u}(y) =\frac{y^{\mathsf T}u}{u^{\mathsf T}u}u.$$

Define the leftover, or residual, as

$$r=y-\operatorname{proj}_{u}(y).$$

The residual is orthogonal to $u$ because

$$\begin{aligned} u^{\mathsf T}r &=u^{\mathsf T}y -\frac{y^{\mathsf T}u}{u^{\mathsf T}u}u^{\mathsf T}u\\ &=u^{\mathsf T}y-y^{\mathsf T}u\\ &=0. \end{aligned}$$

The last step uses symmetry of the real dot product. This is a geometric reason that projection leaves the nearest point on a line: any remaining movement is perpendicular to that line.

The residual is not zero: it is $(-2/5,6/5)$ and still carries a direction and magnitude. “No linear overlap with $u$” means only that the residual has no component along that one direction. It may contain other structure, nonlinear relationships, noise, or signal useful to a later model.

NumPy makes the test explicit

For one-dimensional arrays, current NumPy documentation defines numpy.dot as the vector inner product. numpy.linalg.norm computes the Euclidean length when used with a one-dimensional array. The following function makes the shape contract and numerical tolerance visible:

import numpy as np


def is_orthogonal(x, y, *, atol=1e-12):
    x = np.asarray(x, dtype=float)
    y = np.asarray(y, dtype=float)
    if x.ndim != 1 or y.ndim != 1 or x.shape != y.shape:
        raise ValueError('x and y must be 1-D vectors with the same shape')

    dot = float(np.dot(x, y))
    scale = max(1.0, np.linalg.norm(x) * np.linalg.norm(y))
    return bool(np.isclose(dot, 0.0, atol=atol * scale, rtol=0.0))


u = np.array([3.0, 1.0])
v = np.array([-1.0, 3.0])
residual = np.array([-0.4, 1.2])

print(np.dot(u, v))
print(is_orthogonal(u, v))
print(np.dot(u, residual))
print(is_orthogonal(u, residual))

The output is

0.0
True
-2.220446049250313e-16
True

The tolerance is not a replacement for the definition. It acknowledges that floating-point arithmetic may leave a tiny signed remainder where exact arithmetic would give zero; a platform may display a slightly different value. NumPy’s isclose documentation also warns that its default absolute tolerance can be inappropriate near zero, so a real pipeline should choose a scale-aware threshold for its data. The explicit shape check prevents a coincidental broadcast or an unintended matrix operation from being mistaken for a pairwise vector test.

Orthogonality, correlation, and independence ask different questions

An inner product compares two fixed vectors in a chosen coordinate space. A correlation coefficient describes how two random variables or observed columns move together after centering and scaling. The connection is precise for a finite dataset, but only after stating the data convention.

Suppose $x_c$ and $y_c$ are two centered feature columns with $m$ observations. Using the population normalization for this finite sample,

$$\operatorname{cov}(x,y)=\frac{1}{m}x_c^{\mathsf T}y_c.$$

If both columns have nonzero variance, their centered dot product is zero exactly when their empirical Pearson covariance, and therefore their Pearson correlation, is zero. The sample convention with denominator $m-1$ has the same zero numerator. That says “no measured linear co-movement in this sample.” It does not say “the features are unrelated in every possible way.”

Here is a small counterexample. Let

$$x=(-1,0,1),\qquad y=(1,-2,1).$$

Both means are zero, and

$$x^{\mathsf T}y=(-1)(1)+0(-2)+1(1)=-1+1=0.$$

The centered columns are orthogonal and have zero empirical correlation. Yet the second column is determined by the first through

$$y=3x^2-2$$

coordinate by coordinate. The relationship is nonlinear, not absent. This is why an AI pipeline that removes or rotates correlated directions can reduce linear redundancy while still leaving nonlinear interactions for later layers. The current scikit-learn feature-selection documentation draws the same boundary in implementation terms: F-test methods estimate linear dependency, while mutual-information methods can detect broader statistical dependency and need more data for accurate estimation.

“Independent” is a probability statement about a joint distribution. A zero inner product between two representation vectors—or even a zero correlation measured on one dataset—is not enough to claim independence. Always name the space, centering convention, sample, and relationship you actually tested.

For AI systems, this distinction gives orthogonality a modest but useful role. If centered feature directions are orthogonal, a linear readout cannot obtain a nonzero score from one direction by projecting the other onto it. If a residual component is orthogonal to a fitted direction, that component is cleanly separated from that one linear contribution. Neither statement guarantees that a nonlinear layer, a different metric, or a different dataset will preserve the same separation.

The next step in the curriculum is Lesson 046: Orthonormal Bases Simplify Coordinates. It adds unit length to orthogonal directions, so inner products can be read directly as coordinates. The Math for AI course hub keeps that upcoming lesson and its prerequisite sequence visible while the English source is prepared for its later Translation Set.

Check your understanding

Question 1

For u = (3, 1) and v = (−1, 3), compute the dot product, both Euclidean norms, and the angle.

Show the step-by-step solution

First multiply matching coordinates: $u^{\mathsf T}v=3(-1)+1(3)=-3+3=0$. Next, $\lVert u\rVert_2=\sqrt{3^2+1^2}=\sqrt{10}$ and $\lVert v\rVert_2=\sqrt{(-1)^2+3^2}=\sqrt{10}$. Both vectors are nonzero, so $\cos(\theta)=0/(\sqrt{10}\sqrt{10})=0$. Therefore $\theta=\arccos(0)=\pi/2=90^\circ$, which is the right-angle interpretation of this zero inner product.

Question 2

Test whether a = (2, −1) and b = (1, 2) are orthogonal, and explain why the result is not caused by either vector being zero.

Show the step-by-step solution

The coordinate products are $2(1)=2$ and $(-1)(2)=-2$. Their sum is $a^{\mathsf T}b=2-2=0$, so the vectors pass the orthogonality test. Their norms are $\lVert a\rVert_2=\sqrt{5}$ and $\lVert b\rVert_2=\sqrt{5}$, both strictly positive. Thus these are two nonzero perpendicular vectors; the zero came from cancellation between coordinates.

Question 3

If x ⊥ y, what happens to the inner product after replacing x by −4x and y by 7y?

Show the step-by-step solution

Bilinearity gives $(-4x)^{\mathsf T}(7y)=(-4)(7)x^{\mathsf T}y=-28x^{\mathsf T}y$. Since $x\perp y$, the original product is $0$, so the new product is $-28(0)=0$. The vectors may become longer and one arrow may reverse, but the zero relationship, and therefore perpendicularity under the same dot product, remains.

Question 4

Using u = (3, 1), classify w = (6, 2) and z = (2, 3) by the sign of their dot products with u.

Show the step-by-step solution

For $w$, $u^{\mathsf T}w=3(6)+1(2)=20>0$, and $w=2u$, so the vectors are aligned with angle $0^\circ$. For $z$, $u^{\mathsf T}z=3(2)+1(3)=9>0$; the positive sign indicates an acute angle, but $z$ is not a scalar multiple of $u$, so the angle is oblique rather than zero. A positive dot product therefore means same broad side of the geometry, not necessarily identical direction.

Question 5

Project y = (5, 3) onto u = (3, 1), then calculate the residual and verify its orthogonality.

Show the step-by-step solution

First compute $y^{\mathsf T}u=5(3)+3(1)=18$ and $u^{\mathsf T}u=3^2+1^2=10$. The projection coefficient is $18/10=9/5$, so $\operatorname{proj}_u(y)=(9/5)(3,1)=(27/5,9/5)$. Subtracting gives $r=(5,3)-(27/5,9/5)=(-2/5,6/5)$. Finally, $u^{\mathsf T}r=3(-2/5)+1(6/5)=-6/5+6/5=0$. The residual is not zero; it simply has no component along $u$.

Question 6

Why can a direct NumPy test using np.dot(x, y) == 0 misclassify a pair of floating-point vectors, and what does the lesson's function do instead?

Show the step-by-step solution

Decimal calculations are represented with finite-precision floating-point numbers, so arithmetic that is exactly zero on paper can leave a tiny value such as $10^{-15}$. An exact equality test would then return false. The function first requires one-dimensional arrays with the same shape, computes the dot product and both norms, and calls np.isclose with an explicit, scale-aware absolute tolerance and rtol=0. The tolerance should be chosen for the data and precision; it should not hide a large dot product or a shape error.

Question 7

For x = (−1, 0, 1) and y = (1, −2, 1), show that the centered vectors are orthogonal but not independent in the ordinary statistical sense.

Show the step-by-step solution

The means are both zero, so these vectors are already centered. Their dot product is $(-1)(1)+0(-2)+1(1)=-1+1=0$, which gives zero empirical covariance and, because both have nonzero variance, zero Pearson correlation. But the coordinates obey $y=3x^2-2$ exactly, so observing $x$ determines $y$. That deterministic nonlinear relationship is dependence; zero linear correlation is not a general independence test.

Question 8

A model says that a residual is ‘uninformative’ because it is orthogonal to a fitted direction. What part of that statement is valid, and what part is too strong?

Show the step-by-step solution

If $r$ is the residual after projecting onto direction $u$, then $u^{\mathsf T}r=0$ is valid: a linear score along $u$ cannot recover a component of $r$ by projection onto that same direction. The conclusion that $r$ is uninformative is too strong. The residual can have nonzero norm, contain another linear direction, encode nonlinear structure, or carry signal useful to a later layer. Orthogonality removes one specified linear overlap, not all information.

Sources

1. Mathematics for Machine Learning book PDF, Section 3.4
2. NumPy documentation: numpy.dot
3. NumPy documentation: numpy.linalg.norm
4. NumPy documentation: numpy.isclose
5. scikit-learn documentation: Feature selection