Kernel and Image Show What a Linear Layer Loses and Can Reach

A model’s projection can turn two different feature vectors into the same representation. Once that happens, no later layer can tell those inputs apart from that representation alone. Two ideas make the effect precise: the kernel identifies input changes a linear mapping erases, while the image describes the outputs it can produce.

By the end, you will be able to find both sets for a small matrix and explain what they reveal about information loss and representable outputs.

Learning outcome: Find simple null-space and image examples and interpret them.

This lesson connects the rank of a matrix from Lesson 033 with the matrix representation of a linear mapping from Lesson 035. Section 2.7.3 of Mathematics for Machine Learning introduces the image and kernel of a linear mapping. The projection, calculation, AI interpretation, visual, and questions below are original.

One projection can erase a direction and restrict its outputs

Consider the mapping $P: \mathbb{R}^3\to\mathbb{R}^3$ represented by

It preserves the first two coordinates and sets the third to zero. The kernel of a linear mapping $T:V\to W$ is the set of inputs in $V$ that map to the zero vector in $W$ . Its image is the set of outputs in $W$ that the mapping can produce from some input. These sets belong to different spaces: the kernel is about inputs; the image is about outputs.

Here, kernel means the null space of a linear map. Machine learning also uses “kernel” for functions that compare examples in kernel methods; that is a separate concept.

For $P$ , an input reaches zero exactly when $x=0$ and $y=0$ , while $z$ can be anything. So the kernel is the $z$ -axis:

\ker(P)=\{(0,0,z):z\in\mathbb{R}\} =\operatorname{span}\{(0,0,1)\}.

Every output has the form $(x,y,0)$ , and each such vector is reachable by using the input $(x,y,0)$ . Therefore the image is the $xy$ -plane:

\operatorname{Im}(P)=\{(x,y,0):x,y\in\mathbb{R}\} =\operatorname{span}\{(1,0,0),(0,1,0)\}.

The figure separates these two questions. The vertical kernel direction marks input variation that vanishes; the shaded image plane marks the outputs that remain available.

One map, two different subspacesP(x, y, z) = (x, y, 0): changing z moves the input along the kernel, while every output stays in the xy-plane.

Input space ℝ³

x = (2, 1, 0)
x + k = (2, 1, 2)
k = (0, 0, 2)

Reachable outputs: Im(P)

Every output has third coordinate 0

P(x) = P(x + k) = (2, 1, 0)

Kernel vectors explain which inputs collapse together

Take $x=(2,1,0)$ and add the kernel vector $k=(0,0,2)$ . The inputs differ, but their mapped values agree:

So the kernel is not a list of inputs the matrix ignores individually. It is the collection of directions along which an input can move without changing the result. If a model only receives the projected vector, it cannot recover where along those directions the original input started.

The image is a limit on what downstream code can receive

The same $P$ has rank two because its image has two independent directions. Its kernel also has dimension one. These numbers add to the dimension of the input space:

\underbrace{\dim(\ker(P))}_{\text{nullity }1} +\underbrace{\dim(\operatorname{Im}(P))}_{\text{rank }2} =\underbrace{\dim(\mathbb{R}^3)}_{3}.

This is the rank-nullity theorem for a linear mapping from a finite dimensional input space. It connects the number of independent output directions to the number of independent input directions collapsed to zero. For an $m\times n$ matrix acting on column vectors, the image is the span of its columns, the kernel solves $Ax=0$ , and rank plus nullity equals $n$ , the number of input coordinates.

That last number matters when reading shapes in code. NumPy’s @ operator is matrix multiplication: an $m\times n$ matrix times an $n$ -entry vector produces an $m$ -entry vector. For this projection, a short calculation confirms the shared output and the vanishing kernel direction:

import numpy as np

P = np.diag([1, 1, 0])
x = np.array([2, 1, 2])
k = np.array([0, 0, 2])

print(P @ x)        # [2 1 0]
print(P @ (x - k))  # [2 1 0]
print(P @ k)        # [0 0 0]

The same distinction helps diagnose a learned projection

A learned weight matrix also maps one feature vector to another. PyTorch’s torch.nn.Linear applies an affine map, $y=xA^\mathsf{T}+b$ , to row-oriented inputs; setting bias=False leaves the matrix map alone. In column-vector notation the same operation is $Au+b$ . The weight map determines which input differences disappear: if $A(u-v)=0$ , then the two inputs produce the same output because any shared bias cancels. The image of the weight map describes output directions available from the weights; with a bias, the full output set is the translated subspace $b+\operatorname{Im}(A)$ (PyTorch documentation).

Kernel and image answer separate debugging questions. A nonzero kernel means some distinct feature vectors become indistinguishable at this layer. A restricted image means some output feature combinations cannot be produced at all. A map from $\mathbb{R}^3$ to $\mathbb{R}^2$ can have a nontrivial kernel while reaching every point in its two-dimensional output space. Conversely, a map from $\mathbb{R}^2$ into $\mathbb{R}^3$ can preserve all input distinctions while its outputs lie in a plane. Neither question can be answered from the matrix’s output shape alone; rank and the actual subspaces tell you.

The next step is to ask which directions are redundant before compressing a representation. When that choice is learned rather than specified, inspecting the weight map’s rank, kernel, and image reveals what the downstream model can still distinguish and express.