Transpose for AI: Reorienting Token and Feature Relationships

Suppose a batch contains two tokens, and each token is represented by three features. You store the data as two rows and three columns. A later operation, however, is written to compare features across tokens. Did the data change, or does the operation need a different orientation?

The answer is often a transpose: the same six values are read with rows and columns exchanged. That small operation is a major shape-debugging tool in machine learning because axis meaning is part of a tensor’s contract.

By the end of this lesson, you will be able to transpose a matrix, predict its output shape, explain what swaps semantically, and choose between token-by-feature and feature-by-token layouts without losing track of the underlying values.

Learning outcome: Transpose a matrix and explain what swaps semantically.

Lesson 18: Matrices organize linear rules introduced rows and columns as meaningful positions inside a matrix. The examples and visual here are original applications of the transpose notation developed in Section 2.2 of Mathematics for Machine Learning.

A matrix stores relationships along its axes

Start with a matrix $X$ whose shape is $2\times3$:

$$X = \begin{bmatrix} 2 & -1 & 0\\ 4 & 3 & 1 \end{bmatrix}.$$

The first dimension has length $2$, so there are two rows. The second has length $3$, so there are three columns. Shape alone tells us the dimensions; labels tell us what those dimensions mean. In this lesson, let each row be one token and each column be one feature:

	$f_1$	$f_2$	$f_3$
$t_1$	2	−1	0
$t_2$	4	3	1

The entry $X_{2,3}=1$ means “row 2, column 3”: the third feature of the second token. The indices are not decorative. If code later treats columns as tokens, the numbers may still look plausible while the model is reading the wrong relationships.

Definition: transpose swaps positions, not values

The transpose of a matrix $X$ is written $X^{\mathsf T}$. Its defining rule is

$$\left(X^{\mathsf T}\right)_{j,i}=X_{i,j}.$$

In words, the entry at row $i$, column $j$ moves to row $j$, column $i$. Therefore, if $X$ has shape $m\times n$, then $X^{\mathsf T}$ has shape $n\times m$:

$$X\in\mathbb{R}^{m\times n} \quad\Longrightarrow\quad X^{\mathsf T}\in\mathbb{R}^{n\times m}.$$

The transpose does not negate, sort, scale, or recalculate any entry. It reorients the relationship between an entry and its row and column labels.

X: tokens by features

Two token rows, with one feature in each column.

Original matrix X, shape 2 by 3

Token	f₁	f₂	f₃
t₁	2	−1	0
t₂	4	3	1

Token → rows · Feature → columns

↔

Xᵀ: features by tokens

Three feature rows, with one token in each column.

Transposed matrix Xᵀ, shape 3 by 2

Feature	t₁	t₂
f₁	2	4
f₂	−1	3
f₃	0	1

Feature → rows · Token → columns

Cell correspondence:X[2, 3] = 1 becomes Xᵀ[3, 2] = 1: the value keeps its identity while its coordinates swap.

Inspect one cell step by step

The value in the second token and third feature moves to the third row and second column after transposition.

Before

X[2, 3] = 1

After

Xᵀ[3, 2] = 1

The visual uses labels rather than color as the primary cue. Its two semantic tables are also the complete fallback: a reader can verify every value, dimension, and axis meaning without JavaScript. The disclosure is a native keyboard-operable control, and the component has no continuing motion to interfere with reduced-motion preferences.

Worked example: transpose a $2\times3$ matrix

Use the definition one position at a time. The first row of $X^{\mathsf T}$ comes from the first column of $X$:

$$\left(X^{\mathsf T}\right)_{1,1}=X_{1,1}=2, \qquad \left(X^{\mathsf T}\right)_{1,2}=X_{2,1}=4.$$

The second row comes from the second column:

$$\left(X^{\mathsf T}\right)_{2,1}=X_{1,2}=-1, \qquad \left(X^{\mathsf T}\right)_{2,2}=X_{2,2}=3.$$

The third row comes from the third column:

$$\left(X^{\mathsf T}\right)_{3,1}=X_{1,3}=0, \qquad \left(X^{\mathsf T}\right)_{3,2}=X_{2,3}=1.$$

Putting those entries together gives

$$X^{\mathsf T}= \begin{bmatrix} 2 & 4\\ -1 & 3\\ 0 & 1 \end{bmatrix}, \qquad X^{\mathsf T}\in\mathbb{R}^{3\times2}.$$

Two checks catch most hand errors. First, the shape is reversed from $2\times3$ to $3\times2$. Second, reading $X^{\mathsf T}$ down a column should recover a row of $X$: its first column is $[2,4]^{\mathsf T}$, which was the first row of $X$.

Why orientation matters in AI data

A row-wise batch makes one common operation easy. If $X\in\mathbb{R}^{b\times d}$ stores $b$ examples with $d$ features, then a feature-to-output weight matrix $W\in\mathbb{R}^{k\times d}$ is commonly applied as

$$Y=XW^{\mathsf T}, \qquad (b\times d)(d\times k)=(b\times k).$$

Here the transpose is not changing the learned numbers. It makes the weight’s feature axis line up with the batch matrix’s feature columns. The output has $b$ rows, one for each example, and $k$ columns, one for each output feature.

The same data can be stored feature-first as $X^{\mathsf T}\in\mathbb{R}^{d\times b}$. That view is useful when the question is “how does each feature vary across examples?” but it is not interchangeable with the row-batch view. Always transpose back, or write the product in the feature-first convention, before passing the array to code that expects examples in rows.

$X^{\mathsf T}X$ and $XX^{\mathsf T}$ ask different questions

The transpose often appears next to a matrix product. Their shapes reveal what is being compared:

$$X^{\mathsf T}X: \quad (d\times b)(b\times d)=(d\times d),$$

so each entry compares two features across the $b$ examples. By contrast,

$$XX^{\mathsf T}: \quad (b\times d)(d\times b)=(b\times b),$$

so each entry compares two examples across their $d$ features. Neither product is “more transposed” than the other; they answer different relationship questions.

For the lesson’s matrix,

$$X^{\mathsf T}X= \begin{bmatrix} 10 & 2 & 11\\ 2 & 4 & -2\\ 11 & -2 & 17 \end{bmatrix} \quad\text{and}\quad XX^{\mathsf T}= \begin{bmatrix} 6 & -1\\ -1 & 25 \end{bmatrix}.$$

The first is $3\times3$ because there are three features. The second is $2\times2$ because there are two tokens. The numbers are related, but their indices have different meanings.

Implement the swap in NumPy

For a small teaching example, an explicit loop mirrors the definition. It allocates the reversed shape and copies matrix[row, column] to result[column, row]:

import numpy as np

X = np.array([[2, -1, 0],
              [4,  3, 1]])

def transpose_by_loops(matrix):
    rows, columns = matrix.shape
    result = np.empty((columns, rows), dtype=matrix.dtype)
    for row in range(rows):
        for column in range(columns):
            result[column, row] = matrix[row, column]
    return result

X_T_by_loops = transpose_by_loops(X)
X_T = X.T

np.testing.assert_array_equal(X_T_by_loops, X_T)
np.testing.assert_array_equal(X_T.T, X)

feature_relationships = X.T @ X
token_relationships = X @ X.T
print(X_T)
print(feature_relationships.shape, token_relationships.shape)

The loop is valuable for understanding, not for production speed. NumPy’s ndarray.T is the concise two-dimensional spelling of transpose, while np.transpose generalizes to arrays with more axes. The official NumPy documentation notes an important boundary: transposing a one-dimensional array leaves its shape unchanged. A vector with shape (3,) has no separate row and column axes to exchange; use vector[:, np.newaxis] for an explicit (3, 1) column or vector[np.newaxis, :] for a (1, 3) row.

When the next operation is a matrix product, keep the operators distinct: @ means matrix multiplication, whereas * is elementwise multiplication. For example, X.T @ X contracts the token axis and returns a (3, 3) matrix; X.T * X.T only works when the shapes match and returns elementwise products. The NumPy transpose reference, .T reference, and matrix-product reference are the implementation sources for these distinctions.

Boundary case: transpose is not an inverse

It is tempting to call $X^{\mathsf T}$ the operation that “undoes” $X$. That is wrong in general. A transpose only swaps axes. An inverse, when it exists, is a different matrix $X^{-1}$ that satisfies

$$X^{-1}X=I \quad\text{and}\quad XX^{-1}=I.$$

An inverse requires a square, nonsingular matrix. A $2\times3$ matrix such as the lesson’s $X$ cannot have an ordinary two-sided inverse because its shape does not even match a square identity. Even for a square matrix, $X^{\mathsf T}$ is an inverse only in special cases, such as an orthogonal matrix.

There is a second boundary case in code. The expression X.T is unambiguous for a two-dimensional matrix, but vector.T does nothing to a one-dimensional NumPy array. If a model expects one feature column, make that dimension explicit and check array.shape before the matrix product. A shape error is often useful evidence that an axis convention was left implicit.

The durable debugging sequence is:

1. State what each axis means.
2. Predict the new shape before transposing.
3. Verify one or two index mappings by hand.
4. Check the next operator’s shape contract.
5. Confirm that the resulting rows and columns still match the model’s convention.

Lesson 22 will use identity and inverse transformations to discuss when a linear operation really can be undone. The transpose is the careful bridge: it changes orientation so the next relationship is expressed with compatible axes, but it does not promise reversibility.

Check your understanding

Question 1

If A has shape 4 × 7, what shape does Aᵀ have, and where does the entry A[3, 6] move?

Show the step-by-step solution

Step 1: reverse the dimensions. Transposition changes $(m,n)$ into $(n,m)$, so a $4\times7$ matrix becomes $7\times4$.

Step 2: swap the indices. The defining rule is $(A^{\mathsf T})_{j,i}=A_{i,j}$. Therefore the entry at row 3, column 6 moves to row 6, column 3.

Step 3: interpret the result. The value does not change; only the row and column positions, and therefore their labels, exchange.

Question 2

Transpose M = [[5, −2, 1], [0, 4, 3]] by hand and state the shape of the result.

Show the step-by-step solution

Step 1: count the original axes. $M$ has two rows and three columns, so its transpose must have three rows and two columns.

Step 2: copy columns as rows. The first column $[5,0]$ becomes row one; the second column $[-2,4]$ becomes row two; the third column $[1,3]$ becomes row three.

Step 3: write the result.

$$M^{\mathsf T}=\begin{bmatrix}5&0\\-2&4\\1&3\end{bmatrix}, \qquad M^{\mathsf T}\in\mathbb{R}^{3\times2}.$$

Question 3

A matrix has rows for documents and columns for vocabulary features. What do the rows and columns mean after transposition?

Show the step-by-step solution

Step 1: name the original axes. Before transposition, each row is one document and each column is one vocabulary feature.

Step 2: apply the index rule. Every document-feature value moves from $(document, feature)$ to $(feature, document)$.

Step 3: state the new meaning. After transposition, each row is one vocabulary feature and each column is one document. The values are unchanged, but a row-wise operation now aggregates across documents rather than across features. That is why axis meaning must travel with the array.

Question 4

Why does (Aᵀ)ᵀ = A, and what does this identity not imply?

Show the step-by-step solution

Step 1: swap once. The first transpose sends entry $(i,j)$ to $(j,i)$.

Step 2: swap again. The second transpose sends that same entry from $(j,i)$ back to $(i,j)$. Every value returns to its original position.

Step 3: keep the distinction. This identity says that transpose is an involution: applying the same orientation swap twice returns the matrix. It does not say that $A^{\mathsf T}$ is an inverse or that the first operation undoes a learned transformation in the inverse-matrix sense.

Question 5

For X with shape (8, 5), what are the shapes of XᵀX and XXᵀ, and what does each index pair compare?

Show the step-by-step solution

Step 1: transpose the shape. $X^{\mathsf T}$ has shape $(5,8)$.

Step 2: multiply the first order. $(5,8)(8,5)\to(5,5)$, so $X^{\mathsf T}X$ compares pairs of the five features across the eight examples.

Step 3: multiply the second order. $(8,5)(5,8)\to(8,8)$, so $XX^{\mathsf T}$ compares pairs of the eight examples across their five features. Reversing the order changes the question, not just the notation.

Question 6

In NumPy, why does vector.T keep a vector with shape (3,) at shape (3,), while vector[:, np.newaxis] has shape (3, 1)?

Show the step-by-step solution

Step 1: inspect the number of axes. Shape (3,) describes one axis of length three. There is no distinct row axis and column axis for .T to swap.

Step 2: read NumPy’s behavior. The one-dimensional transpose is an unchanged view, so vector.T.shape is still (3,).

Step 3: add the intended axis explicitly. vector[:, np.newaxis] inserts a second axis after the three entries, giving (3, 1). That is an explicit column vector and can participate in a matrix product with the intended shape.

Question 7

A learner says that Xᵀ is the inverse of X because applying transpose twice returns X. What is the precise correction?

Show the step-by-step solution

Step 1: separate the identities. $(X^{\mathsf T})^{\mathsf T}=X$ means that two axis swaps restore the original positions.

Step 2: state the inverse requirement. An inverse must multiply with $X$ to produce an identity matrix on the appropriate side.

Step 3: use the boundary case. The lesson’s $2\times3$ matrix is not square, so it cannot have an ordinary two-sided inverse at all. Even a square matrix need not equal its transpose inverse. Transpose is an orientation operation; inverse is a reversibility property.

Question 8

A batch X has shape (32, 768) and a weight matrix W has shape (256, 768). Why is X @ W invalid, and which transpose gives the usual row-batch output?

Show the step-by-step solution

Step 1: check the attempted product. The shapes are $(32,768)$ and $(256,768)$. Matrix multiplication would need the left columns, 768, to equal the right rows, 256. They do not, so X @ W is invalid.

Step 2: transpose the weight. $W^{\mathsf T}$ has shape $(768,256)$.

Step 3: check the corrected product. $(32,768)(768,256)\to(32,256)$. The 32 rows remain examples and the 256 columns are output features. The transpose fixes the axis convention, not the values of the learned weights.

Sources

1. Mathematics for Machine Learning companion website
2. Mathematics for Machine Learning book PDF
3. NumPy documentation: numpy.transpose
4. NumPy documentation: ndarray.T
5. NumPy documentation: numpy.matmul