Identity and Inverse Transformations: Undoing a Feature Map

Suppose a preprocessing step mixes two features before they enter an AI model. Later, you want to recover the original coordinates for debugging, a report, or another model. How can you tell whether the transformation is genuinely reversible, and how can you undo it without making a fragile numerical choice?

By the end of this lesson, you will be able to recognize identity behavior, compute and verify a small matrix inverse, distinguish an inverse from a transpose, identify singular and nearly singular cases, and explain why a linear system is usually solved directly instead of by explicitly forming an inverse.

Learning outcome: Recognize identity behavior and verify a small matrix inverse.

Lesson 20: Matrix-matrix products compose transformations and batches showed that a product can compose linear maps. The official Mathematics for Machine Learning companion develops identity, inverse, and transpose operations in Section 2.2 of the book PDF, then connects them to systems of equations in Section 2.3. The notation here follows that path, but the numbers, visual, preprocessing scenario, code, and questions are original.

Identity means “leave the coordinates alone”

For a two-dimensional column vector, the identity matrix is

Multiplying by $I_2$ leaves every coordinate unchanged. If $\boldsymbol{x}=[x_1,x_2]^{\mathsf T}$, then

The diagonal ones preserve each coordinate, while the off-diagonal zeros stop one coordinate from contributing to the other. The identity is the matrix version of the number $1$: multiplying by it changes nothing, but it gives us a precise target for checking whether another matrix has been undone.

The two sides matter. A matrix product is ordered, so a useful inverse must restore the identity when multiplied on either side, not merely make one selected vector look correct.

An inverse reverses a transformation

Let $A$ be a square matrix. An inverse, written $A^{-1}$, is a matrix that satisfies

If a feature vector first becomes $\boldsymbol{z}=A\boldsymbol{x}$, then multiplying by $A^{-1}$ recovers the input:

This is a cancellation rule for matrices. It is not a claim that every matrix has an inverse. A matrix must preserve enough independent directions for an inverse to exist; we will see what fails when it does not.

The visual uses

The unit square becomes a parallelogram under $A$ and returns to the same corners under $A^{-1}$. The table is the static, text-readable record of each corner correspondence. The shape stays one-to-one because the determinant of this $A$ is $1$, not zero.

Worked example: invert a 2 × 2 matrix

For

the determinant is

When this number is nonzero, the 2 × 2 inverse formula is

The formula has two checks built into it: swap the diagonal entries, negate the off-diagonal entries, and divide by the determinant. The nonzero determinant is what makes that division legal.

We can also test the inverse by transforming one feature vector. Let

The forward map gives

The inverse returns it:

The matrix product check is stronger than this one-vector check. Many incorrect matrices can happen to undo one particular vector; only the identity equations show that the transformation has been reversed for every vector in the space.

An inverse is not a transpose

The transpose $A^{\mathsf T}$ swaps rows and columns:

That is an axis reorientation. The inverse solves the different requirement $AA^{-1}=I$. They can be equal in special cases, but one operation cannot be substituted for the other in general.

For the worked matrix, $A$ happens to be symmetric, so $A^{\mathsf T}=A$. But its inverse is

which is clearly not $A$. Symmetry alone does not make a matrix orthogonal or make its transpose an inverse.

Here is a nonsymmetric counterexample:

The transpose preserves the entries but changes their orientation. The inverse changes signs and combines coordinates so that the two matrix products become the identity. Only when $Q^{\mathsf T}Q=I$—for example, for a pure rotation or reflection—does $Q^{-1}=Q^{\mathsf T}$.

AI application: reverse a learned feature coordinate change

Many pipelines work with a centered feature vector rather than the raw feature values. Let $\boldsymbol{x}\in\mathbb{R}^2$ be the raw two-feature vector and $\boldsymbol{\mu}$ be a fixed mean vector. A linear preprocessing step can be written as

The matrix $A$ mixes and rescales the centered coordinates. This can be useful when a later model works better in a rotated or otherwise re-expressed feature system. If $A$ is invertible and the same $\boldsymbol{\mu}$ is retained, the raw vector can be recovered with

The inverse is not a magic undo button for an entire preprocessing pipeline. It reverses this particular linear map. A pipeline that then clips values, drops a feature, rounds to integers, or applies a non-injective operation has discarded information that no matrix inverse can reconstruct.

For example, take

First center the feature vector:

The transformed coordinates are $\boldsymbol{z}=[8,5]^{\mathsf T}$, as in the worked example. Reversing them gives

The practical debugging habit is to record the axis meanings, the mean used for centering, and the exact matrix version together. An inverse from a newer preprocessing fit may be mathematically valid and still be the wrong inverse for an older batch.

Singular transformations lose distinctions

Consider

Its determinant is

The second row is twice the first, so $S$ has only one independent direction. It also maps different inputs to the same output:

Once those two inputs collide, an output-only observer cannot know which input was used. An inverse would have to send one output to two different vectors, which is impossible for a function. NumPy reports this failure as a LinAlgError when numpy.linalg.inv or numpy.linalg.solve is asked to work with a singular square matrix; see the inverse documentation and solve documentation.

Singularity is not only a software error. It can mean that a feature map has collapsed a direction, two columns are redundant, or a model’s representation does not contain enough information for the requested reconstruction. A pseudoinverse or a least-squares estimate may still be useful for a different question, but it is not the exact two-sided inverse of $S$.

Nearly singular is a numerical warning, not a success

Now change one entry by a very small amount:

Its determinant is approximately $10^{-8}$, so it is technically invertible. But its rows are almost parallel. Small measurement or rounding changes can produce large changes in the recovered vector. A condition number summarizes this sensitivity; in the usual 2-norm, a value near $4\times10^8$ means that many digits of input accuracy may be consumed by the solve. NumPy exposes this diagnostic as numpy.linalg.cond.

The distinction is important:

Determinant alone is not a universal scale-free safety threshold. Multiplying a matrix by a large scalar also changes its determinant, so inspect a condition number, singular values, residuals, and the scale of the application. A nonzero floating-point determinant is evidence of invertibility for that representation, not a guarantee of a trustworthy reconstruction.

Why solve is preferred to building an explicit inverse

The equation

has the mathematical solution $\boldsymbol{x}=A^{-1}\boldsymbol{b}$ when $A$ is invertible. In code, however, if all you need is $\boldsymbol{x}$, forming the full $A^{-1}$ first is unnecessary. numpy.linalg.solve accepts $A$ and $\boldsymbol{b}$ directly and computes the solution of the system; numpy.linalg.inv constructs the inverse matrix itself and warns that ill-conditioned results may be inaccurate.

The preferred implementation pattern is therefore:

import numpy as np

A = np.array([[2.0, 1.0],
              [1.0, 1.0]])
b = np.array([2.0, 1.0])

x_from_solve = np.linalg.solve(A, b)
x_from_inverse = np.linalg.inv(A) @ b

np.testing.assert_allclose(x_from_solve, [1.0, 0.0])
np.testing.assert_allclose(x_from_inverse, x_from_solve)
np.testing.assert_allclose(A @ x_from_solve, b)

Both paths agree for this small, well-conditioned example. The important engineering difference is that solve computes the requested solution without materializing a matrix that you may never use. It also keeps the system-solving algorithm explicit, so the code communicates the real task: solve $A\boldsymbol{x} =\boldsymbol{b}$.

If several right-hand sides share the same $A$, pass them as columns in a matrix $B$ and solve $AX=B$ in one call. If $A$ is singular, solve is the wrong tool for an exact solution; choose a method such as least squares only when its different objective matches the application. Always check a residual:

A small residual in the problem’s units is evidence that the computed vector solves the supplied system. It does not, by itself, prove that the system was well-conditioned or that the matrix represented the right preprocessing step.

The next lesson will make the solving step itself explicit with row operations. For now, identity is the target, inverse is the reversible map, transpose is an axis swap, and conditioning tells us whether a mathematically legal reversal is numerically sensible.

Check your understanding

x = np.linalg.solve(A, b)

np.testing.assert_allclose(A @ x, b)