Matrix-Vector Products for AI: Two Views of One Dense-Layer Calculation

Three feature values enter a dense layer and two scores leave it. The layer’s code can be only one line:

scores = weights @ features

That @ hides two complete weighted sums. It also hides a second, equally correct interpretation: each input feature scales one column of the weight matrix, and the scaled columns add to the same output vector. Knowing both views turns a compact operator into something a reader can calculate and debug.

By the end of this lesson, you will be able to compute a matrix-vector product by rows and by columns, explain why the methods agree, implement both in NumPy, and catch incompatible shapes or an accidental elementwise product.

Learning outcome: Compute a matrix-vector product by rows and by columns.

Lesson 17: Linear Combinations for AI showed how scaled vectors add. Lesson 18: Matrices for AI organized one output rule per row and one input feature per column. Matrix-vector multiplication joins those ideas. The official Mathematics for Machine Learning companion develops compatible matrix products and the column-combination view in Section 2.2 of the book. The ranking example, calculation, code, visual, and assessments below are original.

The shared dimension is the calculation contract

Let $W\in\mathbb{R}^{m\times n}$ be a real matrix with $m$ rows and $n$ columns. Let $\boldsymbol{x}\in\mathbb{R}^{n}$ be a vector with $n$ coordinates. The product $W\boldsymbol{x}$ is defined because the matrix’s column count equals the vector’s coordinate count:

$$\underbrace{W}_{m\times n} \underbrace{\boldsymbol{x}}_{n\times1} = \underbrace{\boldsymbol{y}}_{m\times1}.$$

The two matching $n$ values describe the work done inside the product. The remaining $m$ becomes the output length.

This is not permission to cancel the matching dimensions as if they were ordinary numbers. The dimensions record alignment: $x_j$ must meet column $j$, which places weight $w_{ij}$ and input $x_j$ in the same term.

One input produces two scores, row by row

Suppose a small ranking layer uses three standardized input features:

• $x_1=2$: topic match is two scale units above its reference value;
• $x_2=1$: recency is one scale unit above its reference value; and
• $x_3=-1$: source quality is one scale unit below its reference value.

The feature vector and the two-row weight matrix are

$$\boldsymbol{x}= \begin{bmatrix}2\\1\\-1\end{bmatrix}, \qquad W= \begin{bmatrix} 0.8 & -0.4 & 0.2\\ 0.1 & 0.6 & -0.5 \end{bmatrix}.$$

The dimensions predict a two-coordinate answer:

$$(2\times3)(3\times1)\longrightarrow(2\times1).$$

The negative input $x_3=-1$ makes the negative weight $w_{23}=-0.5$ contribute a positive $0.5$ to $y_2$. A weight’s sign alone does not determine whether its contribution raises or lowers an output; the input’s sign matters too.

Explore the calculation

Row viewColumn view

Use the arrow keys on the slider. Step 3 shows the complete product.

Terms included

3 of 3

Each output row includes the first 3 term(s).

Weight matrix W

0.8-0.40.20.10.6-0.5

×

Input x

21-1

=

Current accumulated output

11.3

Row 1: 1.6 + (-0.4) + (-0.2) = 1

Row 2: 0.2 + 0.6 + 0.5 = 1.3

Numeric fallback: each scaled column and its two output-coordinate contributions.

Column contribution	Contribution to y₁	Contribution to y₂	Scaled column
x₁w₁ (x₁ = 2)	1.6	0.2	2 × [0.8, 0.1]ᵀ = [1.6, 0.2]ᵀ
x₂w₂ (x₂ = 1)	-0.4	0.6	1 × [-0.4, 0.6]ᵀ = [-0.4, 0.6]ᵀ
x₃w₃ (x₃ = −1)	-0.2	0.5	-1 × [0.2, -0.5]ᵀ = [-0.2, 0.5]ᵀ
Total	1	1.3	[1, 1.3]ᵀ

The complete table stays visible in the server-rendered lesson, so the six scalar products and their two totals remain available without JavaScript or motion.

The simulator begins with the complete row calculation already rendered. Its column view does not introduce a different product. It regroups the same six scalar multiplications into three vector contributions. The table preserves all exact values even before the interaction hydrates or when motion is reduced.

Columns reveal what each input contributes

Write the columns of $W$ as vectors $\boldsymbol{w}_1,\boldsymbol{w}_2,\boldsymbol{w}_3\in\mathbb{R}^{2}$:

$$\boldsymbol{w}_1= \begin{bmatrix}0.8\\0.1\end{bmatrix}, \qquad \boldsymbol{w}_2= \begin{bmatrix}-0.4\\0.6\end{bmatrix}, \qquad \boldsymbol{w}_3= \begin{bmatrix}0.2\\-0.5\end{bmatrix}.$$

The input coordinates become coefficients for those columns:

$$W\boldsymbol{x} =x_1\boldsymbol{w}_1+x_2\boldsymbol{w}_2+x_3\boldsymbol{w}_3.$$

For the same example,

$$\begin{aligned} W\boldsymbol{x} &=2\begin{bmatrix}0.8\\0.1\end{bmatrix} +1\begin{bmatrix}-0.4\\0.6\end{bmatrix} -1\begin{bmatrix}0.2\\-0.5\end{bmatrix}\\[4pt] &=\begin{bmatrix}1.6\\0.2\end{bmatrix} +\begin{bmatrix}-0.4\\0.6\end{bmatrix} +\begin{bmatrix}-0.2\\0.5\end{bmatrix}\\[4pt] &=\begin{bmatrix}1.0\\1.3\end{bmatrix}. \end{aligned}$$

The row view answers, “How is output $i$ computed?” The column view answers, “What does input $j$ contribute to every output?” A dense-layer debugger often needs both questions.

Why the two views must agree

The equality follows coordinate by coordinate. The $i$th coordinate of the column combination is

$$\left(x_1\boldsymbol{w}_1+\cdots+x_n\boldsymbol{w}_n\right)_i =x_1w_{i1}+\cdots+x_nw_{in} =\sum_{j=1}^{n}w_{ij}x_j.$$

That final sum is exactly the row definition of $(W\boldsymbol{x})_i$. Since every one of the $m$ coordinates agrees, the resulting vectors are equal. The argument uses only ordinary multiplication and addition of a finite number of real values; no new assumption is hiding between the two diagrams.

NumPy can expose the compressed operation

NumPy’s current matmul documentation defines @ as matrix multiplication and documents the usual result for a two-dimensional array multiplied by a one-dimensional array. The compact operation should come after—not instead of—the shape check:

import numpy as np

weights = np.array([
    [0.8, -0.4,  0.2],
    [0.1,  0.6, -0.5],
], dtype=float)
features = np.array([2.0, 1.0, -1.0])

if weights.shape[1] != features.shape[0]:
    raise ValueError("each matrix row needs one weight per input feature")

direct = weights @ features
by_rows = np.array([
    sum(weight * value for weight, value in zip(row, features))
    for row in weights
])
by_columns = sum(
    (features[j] * weights[:, j] for j in range(weights.shape[1])),
    start=np.zeros(weights.shape[0]),
)

print("direct:", direct)
print("by rows:", by_rows)
print("by columns:", by_columns)
print("all equal:", np.allclose(direct, by_rows)
      and np.allclose(direct, by_columns))

Verified output:

direct: [1.  1.3]
by rows: [1.  1.3]
by columns: [1.  1.3]
all equal: True

The explicit versions are teaching and debugging tools. For ordinary numerical work, weights @ features communicates the intended linear-algebra operation directly and lets NumPy use its optimized implementation.

The output shape deserves attention. weights has shape (2, 3), while features has shape (3,). NumPy treats that one-dimensional second operand as the vector in a matrix-vector product and returns shape (2,), not (2, 1). The values match the column-vector notation; the software representation has simply omitted a singleton axis.

A dense layer adds its bias after the product

The matrix-vector product is the linear part of a dense layer. A layer with a bias uses the affine rule

$$\boldsymbol{z}=W\boldsymbol{x}+\boldsymbol{b}, \qquad \boldsymbol{b}\in\mathbb{R}^{m}.$$

If $\boldsymbol{b}=[0.2,-0.1]^{\mathsf T}$ in the running example, then

$$\boldsymbol{z} =\begin{bmatrix}1.0\\1.3\end{bmatrix} +\begin{bmatrix}0.2\\-0.1\end{bmatrix} =\begin{bmatrix}1.2\\1.2\end{bmatrix}.$$

An activation function, if the model uses one, comes after that affine result. It is not part of the definition of $W\boldsymbol{x}$.

The current torch.nn.Linear documentation stores weights with shape (out_features, in_features), the same row-per-output arrangement used here. PyTorch usually accepts inputs with features on the last axis and presents its rule as $\boldsymbol{x}A^{\mathsf T}+\boldsymbol{b}$. For one input represented as a row, that is the transpose-equivalent form of the column-vector equation $W\boldsymbol{x}+\boldsymbol{b}$; it does not create a different set of weights.

Two multiplication signs can produce two different objects

The most dangerous mistake here can run without an exception. NumPy documents multiply and * as elementwise multiplication with broadcasting. Therefore this code

print(weights * features)

does not compute the two-coordinate layer output. The length-three vector broadcasts across both rows, producing

[[ 1.6 -0.4 -0.2]
 [ 0.2  0.6  0.5]]

with shape (2, 3). Those six cells are the contributions before each row is summed. Calling sum(axis=1) would finish this particular calculation, but @ states the intended contraction directly. The NumPy quickstart makes the same operator distinction: * is elementwise for arrays, while @ performs a matrix product.

An incompatible input fails for a different reason:

bad_features = np.array([2.0, 1.0])
weights @ bad_features

The matrix has three columns but the vector has only two coordinates. NumPy’s documented matmul contract rejects that inner-dimension mismatch. Adding a third arbitrary number to silence the error would be worse than the error; the missing feature and its coordinate meaning must be identified first.

The next product keeps the same alignment rule

A matrix-vector product compresses several output formulas without erasing their structure. Reading by rows reveals one weighted sum per output. Reading by columns reveals one output-vector contribution per input. Shape alignment guarantees that both readings pair the same $w_{ij}$ with the same $x_j$.

Lesson 20 will place several input vectors beside the same weight matrix. The operation becomes matrix-matrix multiplication, but the central discipline survives: match the inner dimension, then decide which outer axes the result keeps. The Math for AI course page preserves that prerequisite path.

Check your understanding

Question 1

Let A have shape 4 × 3 and x have length 3. Is Ax defined, what is its output length, and how many scalar products are added for each output coordinate?

Show the step-by-step solution

The matrix has three columns, and the vector has three coordinates. Those inner dimensions match, so the product is defined:

$$(4\times3)(3\times1)\longrightarrow(4\times1).$$

The four remaining rows determine four output coordinates. For each output, one row supplies three weights. Each weight multiplies the aligned input coordinate, and the three scalar products are added. Thus $A\boldsymbol{x}$ has length $4$, with three products contributing to each coordinate.

Question 2

Compute Bv by rows for B = [[2, −1], [0.5, 3]] and v = [4, 2]ᵀ. Show both weighted sums.

Show the step-by-step solution

The two columns of $B$ match the two coordinates of $\boldsymbol{v}$, so the product is defined and will have two outputs.

For row 1,

$$y_1=(2)(4)+(-1)(2)=8-2=6.$$

For row 2,

$$y_2=(0.5)(4)+(3)(2)=2+6=8.$$

Stacking the row results in order gives

$$B\boldsymbol{v}=\begin{bmatrix}6\\8\end{bmatrix}.$$

Question 3

Recompute the product in Question 2 as a weighted sum of B's columns.

Show the step-by-step solution

First separate the columns:

$$\boldsymbol{b}_1=\begin{bmatrix}2\\0.5\end{bmatrix}, \qquad \boldsymbol{b}_2=\begin{bmatrix}-1\\3\end{bmatrix}.$$

The coordinates of $\boldsymbol{v}=[4,2]^{\mathsf T}$ become their coefficients:

$$\begin{aligned} B\boldsymbol{v} &=4\boldsymbol{b}_1+2\boldsymbol{b}_2\\ &=4\begin{bmatrix}2\\0.5\end{bmatrix} +2\begin{bmatrix}-1\\3\end{bmatrix}\\ &=\begin{bmatrix}8\\2\end{bmatrix} +\begin{bmatrix}-2\\6\end{bmatrix}\\ &=\begin{bmatrix}6\\8\end{bmatrix}. \end{aligned}$$

This matches the row calculation because both views contain the same four scalar products, grouped differently.

Question 4

In the lesson's product, set x₂ to zero while keeping x₁ = 2 and x₃ = −1. Use the column view to calculate the new result and explain what remains aligned.

Show the step-by-step solution

The new input is $[2,0,-1]^{\mathsf T}$. Its second coordinate scales the second column to zero, but the coordinate remains in position 2:

$$\begin{aligned} W\boldsymbol{x} &=2\begin{bmatrix}0.8\\0.1\end{bmatrix} +0\begin{bmatrix}-0.4\\0.6\end{bmatrix} -1\begin{bmatrix}0.2\\-0.5\end{bmatrix}\\ &=\begin{bmatrix}1.6\\0.2\end{bmatrix} +\begin{bmatrix}0\\0\end{bmatrix} +\begin{bmatrix}-0.2\\0.5\end{bmatrix}\\ &=\begin{bmatrix}1.4\\0.7\end{bmatrix}. \end{aligned}$$

Column 2 still corresponds to recency and column 3 still corresponds to source quality. Removing the zero coordinate entirely would shorten the vector and misalign every later feature.

Question 5

A developer runs weights * features with shapes (2, 3) and (3,). Why does the code succeed, what shape does it return, and what operation is still missing?

Show the step-by-step solution

NumPy’s * operator performs elementwise multiplication. A length-three array can broadcast across each of the two matrix rows, so the operation succeeds and returns a (2, 3) array of six products.

For the lesson’s values, those rows are

$$\begin{bmatrix} 1.6 & -0.4 & -0.2\\ 0.2 & 0.6 & 0.5 \end{bmatrix}.$$

The dense-layer output needs one number per row. Summing each row is still missing:

$$\begin{bmatrix} 1.6-0.4-0.2\\ 0.2+0.6+0.5 \end{bmatrix} =\begin{bmatrix}1.0\\1.3\end{bmatrix}.$$

Using weights @ features expresses both the multiplication and the required sum.

Question 6

For a compatible matrix A, compare A0 with the output of a biased layer A0 + b. Why are the answers different?

Show the step-by-step solution

Every coordinate of the zero vector equals zero. In the column view, every column is therefore scaled by zero:

$$A\boldsymbol{0} =0\boldsymbol{a}_1+\cdots+0\boldsymbol{a}_n =\boldsymbol{0}.$$

A biased dense layer performs an additional vector addition after the matrix product:

$$A\boldsymbol{0}+\boldsymbol{b} =\boldsymbol{0}+\boldsymbol{b} =\boldsymbol{b}.$$

The answers differ because $A\boldsymbol{x}$ is a linear map, while adding a nonzero bias makes the complete rule affine. The bias is not part of the matrix-vector product.

Question 7

A stored weight array has shape (3, 2), but the intended layer has three inputs and two outputs. Why does W @ x fail for a length-three input, and what must be verified before transposing W?

Show the step-by-step solution

Under the row-per-output convention, the intended matrix shape is (2, 3): two output rows and three input columns. The observed (3, 2) array has only two columns. Its inner dimension therefore cannot match a length-three input:

$$(3\times2)(3\times1)$$

is undefined because $2\ne3$.

Transposing would produce shape (2, 3), but shape alone is not enough. Before doing so, verify that the observed rows truly represent the three input features, the observed columns truly represent the two output rules, and their orders match the feature and output metadata. If the array already carries a different semantic layout, a blind transpose can preserve the wrong meanings.

Question 8

The lesson's product is [1, 1.3]ᵀ. A dense layer adds b = [0.2, −0.1]ᵀ and then applies ReLU, which replaces negative coordinates with zero. Calculate the pre-activation and final output.

Show the step-by-step solution

First add the bias coordinate by coordinate:

$$\boldsymbol{z} =W\boldsymbol{x}+\boldsymbol{b} =\begin{bmatrix}1.0\\1.3\end{bmatrix} +\begin{bmatrix}0.2\\-0.1\end{bmatrix} =\begin{bmatrix}1.2\\1.2\end{bmatrix}.$$

This $\boldsymbol{z}$ is the pre-activation. ReLU applies separately to each coordinate, using $\operatorname{ReLU}(t)=\max(0,t)$. Both coordinates are positive, so neither changes:

$$\operatorname{ReLU}(\boldsymbol{z}) =\begin{bmatrix}\max(0,1.2)\\\max(0,1.2)\end{bmatrix} =\begin{bmatrix}1.2\\1.2\end{bmatrix}.$$

The order matters: compute the matrix-vector product, add the bias, and only then apply the activation.

Sources

1. Mathematics for Machine Learning companion website
2. Mathematics for Machine Learning book PDF
3. NumPy documentation: numpy.matmul
4. NumPy documentation: numpy.multiply
5. NumPy quickstart: elementwise and matrix products
6. PyTorch documentation: torch.nn.Linear