Linear Mappings Preserve the Combinations AI Models Need

A neural network repeatedly turns one representation into another. To follow what a layer does, it helps to ask two questions: what kinds of vectors can go in, and what kinds can come out? A linear mapping answers these questions while preserving the ways vectors combine. This makes it possible to reason about matrix weights as transformations between spaces, then identify exactly where a network introduces nonlinear behavior.

By the end, you will be able to test a mapping for linearity, state its input and output spaces, and tell the matrix transformation in a neural layer apart from its activation.

Learning outcome: Test linearity and describe the input and output spaces of a mapping.

This lesson builds on the vector-space rules from Lesson 027. Section 2.7 of Mathematics for Machine Learning places linear mappings at the point where vector combinations begin to connect to coordinates and matrices. The calculations, neural-layer example, visual, and assessments here are original.

A transformation should respect how its inputs combine

Vectors can be added and scaled. For two real vector spaces $V$ and $W$ , a mapping $T:V\to W$ is linear when it respects both operations:

Here $u,v\in V$ and $a,b\in\mathbb{R}$ . The result is in $W$ , and the coefficients on the right scale the corresponding outputs.

You can test this one combined rule, or split it into two checks:

T(u+v)=T(u)+T(v), \qquad T(au)=aT(u).

The first says that adding two inputs before the transformation gives the same result as transforming each input and then adding. The second says that scaling an input before the transformation scales its output by the same amount. Both must hold for every valid input and every real scalar, including negative ones.

A linear map carries a sum into the same sum of imagesFor T(x₁, x₂) = (2x₁ + x₂, −x₁ + x₂), the image of e₁ + e₂ is the sum of the images of e₁ and e₂.

Input space ℝ²

e₁ = (1, 0)
e₂ = (0, 1)
e₁ + e₂ = (1, 1)

Output space ℝ²

T(e₁) = (2, −1)
T(e₂) = (1, 1)
T(e₁ + e₂) = (3, 0)

Combination check: T(e₁ + e₂) = T(e₁) + T(e₂)

The three arrows show one concrete addition check. Here, $T(e_1+e_2)=(3,0)$ , and $T(e_1)+T(e_2)=(2,-1)+(1,1)=(3,0)$ . The visual offers a geometric reading; the formula is the general promise that must hold for all vectors and scalars, not only for these two basis vectors.

A matrix tells you which spaces the map connects

Let $A$ be an $m\times n$ matrix, meaning it has $m$ rows and $n$ columns. It multiplies an $n$ -coordinate column vector and produces an $m$ -coordinate column vector:

T: \mathbb{R}^n\to\mathbb{R}^m, \qquad T(x)=Ax.

The domain is the set of inputs allowed by the mapping; the codomain is the output space it promises to land in. For this matrix rule, inputs are in $\mathbb{R}^n$ and outputs are in $\mathbb{R}^m$ . The dimensions need not be equal: a map can expand a vector into more coordinates or compress it into fewer.

Matrix multiplication is linear because it distributes over addition and scalar multiplication. For any vectors $u,v$ of length $n$ and scalars $a,b$ ,

T(au+bv)=A(au+bv)=aAu+bAv=aT(u)+bT(v).

For a small numerical example, use

Map a sum, then compare the sum of the images

A=\begin{bmatrix}2&1\\-1&1\end{bmatrix},\qquad u=\begin{bmatrix}1\\0\end{bmatrix},\qquad v=\begin{bmatrix}0\\1\end{bmatrix}.

First find each image by multiplying the matrix by the input:

T(u)=\begin{bmatrix}2\\-1\end{bmatrix},\qquad T(v)=\begin{bmatrix}1\\1\end{bmatrix}.

Then add the inputs and map the result:

T(u+v)=A\begin{bmatrix}1\\1\end{bmatrix} =\begin{bmatrix}3\\0\end{bmatrix} =\begin{bmatrix}2\\-1\end{bmatrix}+\begin{bmatrix}1\\1\end{bmatrix} =T(u)+T(v).

Scaling works the same way. With $a=-2$ ,

T(-2u)=A\begin{bmatrix}-2\\0\end{bmatrix} =\begin{bmatrix}-4\\2\end{bmatrix} =-2T(u).

Negative values matter: a check using only positive scaling does not establish the full linearity rule.

A bias shifts the origin; an activation bends the rule

In machine-learning code, “linear layer” is familiar shorthand. In mathematics, the details determine whether the operation is linear. PyTorch’s torch.nn.Linear computes $xA^\mathsf{T}+b$ for row-oriented batches and includes the bias $b$ by default. With a nonzero bias, the rule is affine: it preserves combinations whose coefficients sum to one, but generally does not preserve arbitrary scaling or sums. The PyTorch documentation calls this an affine transformation and lets you disable the bias with bias=False (torch.nn.Linear documentation). It still preserves a weighted pair when the weights add to one:

F(\lambda u+(1-\lambda)v) =\lambda Au+(1-\lambda)Av+b =\lambda(Au+b)+(1-\lambda)(Av+b).

The last equality works because $\lambda b+(1-\lambda)b=b$ . Ordinary vector addition uses coefficients $1$ and $1$ , which do not add to one, so the bias does not cancel from $F(u+v)$ .

The zero vector is a quick diagnostic. Every linear map must send zero to zero, because $T(0)=T(0\cdot x)=0\cdot T(x)=0$ . For an affine rule $F(x)=Ax+b$ ,

F(0)=A0+b=b.

If $b\ne0$ , then $F$ cannot be linear. The bias does not make the layer incorrect; it gives the model the freedom to shift its output even when the input is zero.

Neural networks add another operation after the weight transformation. A common pattern is $h=\operatorname{ReLU}(Ax+b)$ , where ReLU replaces each negative coordinate with zero. It is nonlinear. For example, let $r(t)=\max(0,t)$ . Then $r(-1+1)=r(0)=0$ , while $r(-1)+r(1)=0+1=1$ . The addition test fails. PyTorch documents torch.nn.ReLU as this elementwise maximum operation (ReLU documentation).

This difference explains why networks combine these steps. If two layers were only linear maps, their composition would still be one linear map: $B(Ax)=(BA)x$ . More layers would change the matrix, but not create a nonlinear input-output rule. An activation interrupts that collapse and lets the network represent more varied relationships. The weight matrix mixes input features; the activation changes the rule applied to the resulting coordinates.

Inspect the transformation in code

The same calculation can be checked with NumPy. Column vectors make the matrix-on-the-left convention match the derivation above:

import numpy as np

A = np.array([[2, 1],
              [-1, 1]])
u = np.array([1, 0])
v = np.array([0, 1])

def T(x):
    return A @ x
print(T(u + v))              # [3 0]
print(T(u) + T(v))           # [3 0]
print(T(-2 * u))             # [-4  2]
print(-2 * T(u))             # [-4  2]

For a batch in PyTorch, vectors are commonly stored as rows, so the library’s weight matrix is transposed in the displayed formula. The batch dimension stays the same while the final feature dimension changes from in_features to out_features:

import torch
from torch import nn

x = torch.tensor([[1.0, 0.0], [0.0, 1.0]])  # shape: (2 examples, 2 features)
linear_part = nn.Linear(2, 3, bias=False)
activation = nn.ReLU()

pre_activation = linear_part(x)  # shape: (2, 3); a linear map
hidden = activation(pre_activation)  # shape: (2, 3); nonlinear elementwise step

Here the linear map is from each feature vector in $\mathbb{R}^2$ to one in $\mathbb{R}^3$ . This instance omits bias so its first operation is genuinely linear. With the default bias, PyTorch describes that operation as affine. The activation preserves the output shape but fails the vector-combination test. Shape agreement alone cannot tell you whether an operation is linear; the algebraic rule does.

A linear map can still lose information

Linearity does not say that a map is reversible, length-preserving, or even useful for every downstream task. The zero matrix maps every input to the zero vector and still satisfies both linearity rules. A linear map from $\mathbb{R}^3$ to $\mathbb{R}^2$ cannot be one-to-one: at least one direction of input information must disappear. To determine whether information is preserved, you need other ideas such as independence, rank, and invertibility.

When you inspect a neural block, write down the space before and after each operation, then test the rule each operation applies. A matrix multiplication transforms combinations predictably; a bias shifts the origin; an activation changes how combinations behave. That is the distinction to carry into the next lesson, where linear maps get concrete matrix representations.

Check your understanding

Question 1

State the two properties that a mapping must satisfy to be linear. What does a vector-addition check leave untested?