Vector Addition for AI: How Residual Updates Combine Signals

A neural-network layer receives a three-coordinate state

\boldsymbol{x}= \begin{bmatrix}2\\-1\\0.5\end{bmatrix}

and computes an update

\boldsymbol{u}= \begin{bmatrix}-0.5\\1.5\\1\end{bmatrix}.

A residual connection does not choose between the old state and the update. It adds matching coordinates, producing $\boldsymbol{y}=\boldsymbol{x}+\boldsymbol{u}=[1.5,0.5,1.5]^\mathsf{T}$ . The first coordinate moves down, the second moves up, and the third moves up; one rule handles all three changes.

By the end of this lesson, you will be able to add vectors component by component, read the same operation as a displacement or feature update, verify it in NumPy, and reject additions whose shapes happen to work while their coordinate meanings do not align.

Learning outcome: Add vectors componentwise and interpret the resulting displacement or feature mix.

Lesson 14: Vectors as AI Data established that a vector is an ordered representation: coordinate $j$ carries a particular feature or learned signal. Vector addition depends on that ordering. The official Mathematics for Machine Learning companion develops row and column vectors and elementwise matrix addition in Section 2.2. This lesson specializes the rule to vectors; its model-state example, derivation, code, interactive geometry, and assessments are original.

One coordinate rule combines the whole vector

Let $\boldsymbol{a},\boldsymbol{b}\in\mathbb{R}^{n}$ be two real vectors with the same positive dimension $n$ . Write their coordinates as $a_j$ and $b_j$ , where $j$ ranges from 1 through $n$ .

The phrase matching coordinates carries two requirements. First, the vectors need the same dimension. Second, each pair $a_j,b_j$ must refer to the same coordinate system: the same feature order, units, and representation convention. Mathematics states the dimension requirement. A data pipeline must also enforce the semantic one.

Addition does not concatenate two vectors. Concatenating $[a_1,a_2]^\mathsf{T}$ and $[b_1,b_2]^\mathsf{T}$ would make a four-coordinate vector. Adding them keeps two coordinates:

\begin{bmatrix}a_1\\a_2\end{bmatrix} + \begin{bmatrix}b_1\\b_2\end{bmatrix} = \begin{bmatrix}a_1+b_1\\a_2+b_2\end{bmatrix}.

Follow a residual update without skipping a coordinate

Return to the opening vectors. Suppose all three positions belong to the same learned representation space. Compute the sum one row at a time:

Coordinate	Existing state $x_j$	Update $u_j$	Output $y_j=x_j+u_j$
1	2.0	−0.5	1.5
2	−1.0	1.5	0.5
3	0.5	1.0	1.5

A positive update raises a coordinate, a negative update lowers it, and a zero update leaves it unchanged. Those descriptions are arithmetic, not semantic interpretations of what an internal neural coordinate “means.” Learned representations can distribute information across many coordinates, so the sum alone does not reveal a human-readable concept.

The diagonal and the two-step path reach the same point

In two dimensions, a vector can be drawn as an arrow from the origin. To add $\boldsymbol{a}$ and $\boldsymbol{b}$ geometrically, travel along $\boldsymbol{a}$ and then place a copy of $\boldsymbol{b}$ at its head. The endpoint is $\boldsymbol{a}+\boldsymbol{b}$ because the horizontal travel is $a_1+b_1$ and the vertical travel is $a_2+b_2$ .

The order can be reversed. Traveling along $\boldsymbol{b}$ and then a translated copy of $\boldsymbol{a}$ reaches the same endpoint. The two paths form adjacent sides of a parallelogram; its diagonal from the origin is the sum. Move the four coordinates below to test positive, negative, parallel, and cancelling directions.

Interactive vector parallelogram: change both vectors and compare the two-step paths with the diagonal sum.

Choose the horizontal and vertical coordinates

Each slider runs from −4 to 4 in steps of 1 and works with the arrow keys.

Vector a, horizontal coordinate a₁+3

Vector a, vertical coordinate a₂+1

Vector b, horizontal coordinate b₁-1

Vector b, vertical coordinate b₂+2

Current calculation: a = (+3, +1), b = (-1, +2), so a + b = (+2, +3).

Vector a — solid arrowVector b — dashed arrowSum a + b — thick diagonal

Both head-to-tail routes end at the coordinates in the sum row. The dashed translated b arrow starts at a; the dotted translated a arrow starts at b.

Numeric alternative for the current parallelogram
Vector	Horizontal coordinate	Vertical coordinate	Geometric role
Vector a — solid arrow	+3	+1	First displacement and one side
Vector b — dashed arrow	-1	+2	Second displacement and one side
Sum a + b — thick diagonal	+2	+3	Final endpoint and diagonal

The diagram uses two coordinates because a flat page can display them. The component rule does not depend on a picture: for a vector in $\mathbb{R}^{n}$ , add each of the $n$ matching coordinates even when $n$ is too large to draw.

Three algebraic facts follow coordinate by coordinate

Vector addition inherits familiar properties of real-number addition. The reason is local: every vector identity can be checked at coordinate $j$ .

Order does not change the sum. Because $a_j+b_j=b_j+a_j$ for real numbers,

(\boldsymbol{a}+\boldsymbol{b})_j =a_j+b_j =b_j+a_j =(\boldsymbol{b}+\boldsymbol{a})_j.

All coordinates agree, so $\boldsymbol{a}+\boldsymbol{b}=\boldsymbol{b}+\boldsymbol{a}$ . This is commutativity, and the parallelogram shows the same fact as two routes to one endpoint.

Grouping does not change a three-vector sum. For another vector $\boldsymbol{c}\in\mathbb{R}^{n}$ ,

\begin{aligned} ((\boldsymbol{a}+\boldsymbol{b})+\boldsymbol{c})_j &=(a_j+b_j)+c_j\\ &=a_j+(b_j+c_j)\\ &=(\boldsymbol{a}+(\boldsymbol{b}+\boldsymbol{c}))_j. \end{aligned}

This associativity gives the same exact mathematical vector under either grouping. Finite-precision code can accumulate slightly different rounding errors when it changes the order of many additions. Neither mathematical associativity nor numeric closeness grants permission to mix coordinate systems.

The zero vector changes nothing. Let $\boldsymbol{0}=[0,\ldots,0]^\mathsf{T}\in\mathbb{R}^{n}$ . Then

(\boldsymbol{a}+\boldsymbol{0})_j=a_j+0=a_j,

so $\boldsymbol{a}+\boldsymbol{0}=\boldsymbol{a}$ . A residual branch that returns the zero vector makes no numerical change at that addition site.

Residual connections use this exact addition mechanism

The original ResNet paper writes a residual block as $\boldsymbol{y}=\mathcal{F}(\boldsymbol{x})+\boldsymbol{x}$ : a learned branch computes $\mathcal{F}(\boldsymbol{x})$ , while a shortcut carries $\boldsymbol{x}$ to the addition. The authors specify that this is elementwise addition and that the two terms need equal dimensions. When dimensions do not match, their architecture can use a learned projection on the shortcut before adding.

The original Transformer offers another concrete instance. Its encoder applies a residual connection around each sublayer:

\operatorname{LayerNorm} \left(\boldsymbol{x}+\operatorname{Sublayer}(\boldsymbol{x})\right).

In that architecture, the embedding and sublayer outputs use the same $d_{\text{model}}=512$ dimension so the coordinatewise sum is defined. The normalization after the sum is another operation; vector addition itself ends at $\boldsymbol{x}+\operatorname{Sublayer}(\boldsymbol{x})$ .

These examples justify a narrow implementation claim: residual connections combine a compatible input and update with elementwise addition. The equation alone does not prove that every input feature survives unchanged, that each coordinate has an isolated meaning, or that any two same-length arrays are a valid residual pair.

NumPy adds values and broadcasts shapes

NumPy’s add operation adds its arguments elementwise. For two one-dimensional arrays with the same shape, np.add(state, update) and state + update implement the component rule directly:

import numpy as np

state = np.array([2.0, -1.0, 0.5])
update = np.array([-0.5, 1.5, 1.0])

if state.shape != update.shape:
    raise ValueError("A residual update must match the state shape")

combined = np.add(state, update)
print("state shape:", state.shape)
print("update shape:", update.shape)
print("combined:", combined)

The verified output is

state shape: (3,)
update shape: (3,)
combined: [1.5 0.5 1.5]

A batch keeps one state per row. Matching arrays of shape (2, 3) add each update to its corresponding state:

batch = np.array([
    [2.0, -1.0, 0.5],
    [0.0,  3.0, -2.0],
])
updates = np.array([
    [-0.5,  1.5, 1.0],
    [ 1.0, -1.0, 0.5],
])

print("batch shape:", batch.shape)
print("batch sums:\n", batch + updates)

The verified output is

batch shape: (2, 3)
batch sums:
 [[ 1.5  0.5  1.5]
 [ 1.   2.  -1.5]]

The first output row is the worked example. The second comes from $[0,3,-2]^\mathsf{T}+[1,-1,0.5]^\mathsf{T}=[1,2,-1.5]^\mathsf{T}$ .

A successful array operation can still be the wrong addition

NumPy intentionally supports broadcasting: if shapes differ but satisfy its compatibility rules, a smaller array can act across a larger one. Adding a shape (3,) update to the shape (2, 3) batch succeeds and applies the same three values to both rows:

broadcast_update = np.array([0.5, 0.0, -0.5])
result = batch + broadcast_update

print("broadcast shape:", result.shape)
print("broadcast sums:\n", result)

broadcast shape: (2, 3)
broadcast sums:
 [[ 2.5 -1.   0. ]
 [ 0.5  3.  -2.5]]

That may be the intended operation—for example, one shared update for every batch row. It is not addition between two individual vectors of equal shape, and it is dangerous when the extra expansion was accidental. Validate the contract your model intends, not merely the shapes NumPy accepts.

Matching shapes are also insufficient. Consider these arrays:

\begin{aligned} \boldsymbol{p}&=[4,1,0]^\mathsf{T} &&\text{with order }(\texttt{clicks},\texttt{purchases},\texttt{returns}),\\ \boldsymbol{q}&=[2,3,5]^\mathsf{T} &&\text{with order }(\texttt{returns},\texttt{clicks},\texttt{purchases}). \end{aligned}

NumPy returns $[6,4,5]^\mathsf{T}$ , but its first calculation adds clicks to returns. No numeric shape check can recover the missing labels. Reorder one vector into the other’s coordinate convention—or reject the operation—before addition.

Vector addition is now more than a formula: it is a coordinate contract, a geometric path, and the combining operation inside a residual block. The next lesson will control the strength of one vector through scalar multiplication; today’s sum supplies the operation that those scaled signals will eventually join.

Check your understanding

Question 1

Add a = [3, −2, 1]ᵀ and b = [−5, 4, 0.5]ᵀ. Show every coordinate calculation and state the dimension of the result.

Show the step-by-step solution

Both inputs have three coordinates, so their sum—if calculated coordinate by coordinate—will also be in $\mathbb{R}^{3}$ .

\begin{aligned} (\boldsymbol{a}+\boldsymbol{b})_1&=3+(-5)=-2,\\ (\boldsymbol{a}+\boldsymbol{b})_2&=-2+4=2,\\ (\boldsymbol{a}+\boldsymbol{b})_3&=1+0.5=1.5. \end{aligned}

Collect the results in their original order:

\boldsymbol{a}+\boldsymbol{b} =\begin{bmatrix}-2\\2\\1.5\end{bmatrix} \in\mathbb{R}^{3}.

Addition changed the values but neither inserted nor removed a coordinate.

Question 2

For a = (2, 1) and b = (−1, 3), trace the head-to-tail route a then b. Where does it end, and why is that point the parallelogram diagonal?

Show the step-by-step solution

Start at the origin $(0,0)$ . Following $\boldsymbol{a}=(2,1)$ moves to $(2,1)$ . A translated copy of $\boldsymbol{b}=(-1,3)$ then changes the horizontal coordinate by $-1$ and the vertical coordinate by $3$ :

(2,1)+(-1,3)=(2-1,1+3)=(1,4).

Reversing the order gives

(-1,3)+(2,1)=(-1+2,3+1)=(1,4).

The two routes are the two pairs of parallel sides of a parallelogram. Since both reach $(1,4)$ , the arrow from the origin to $(1,4)$ is its diagonal and represents $\boldsymbol{a}+\boldsymbol{b}$ .

Question 3

Explain coordinate by coordinate why a + b = b + a for any a, b ∈ ℝⁿ. Do not rely only on a two-dimensional picture.

Show the step-by-step solution

Choose an arbitrary coordinate $j\in\{1,\ldots,n\}$ . By the definition of vector addition,

(\boldsymbol{a}+\boldsymbol{b})_j=a_j+b_j.

Real-number addition is commutative, so $a_j+b_j=b_j+a_j$ . Applying the vector definition in the reverse direction gives

b_j+a_j=(\boldsymbol{b}+\boldsymbol{a})_j.

Therefore the two sums agree at the arbitrary coordinate $j$ . The argument holds for every coordinate, so the vectors are equal: $\boldsymbol{a}+\boldsymbol{b}=\boldsymbol{b}+\boldsymbol{a}$ . This reasoning works in any finite dimension, including dimensions that cannot be drawn.

Question 4

A residual branch returns [0, 0, 0, 0]ᵀ for a state x ∈ ℝ⁴. Calculate the output x + 0 and name the assumption needed before saying the branch made no meaningful change.

Show the step-by-step solution

Write $\boldsymbol{x}=[x_1,x_2,x_3,x_4]^\mathsf{T}$ . Coordinatewise addition gives

\boldsymbol{x}+\boldsymbol{0} = \begin{bmatrix} x_1+0\\x_2+0\\x_3+0\\x_4+0 \end{bmatrix} = \begin{bmatrix}x_1\\x_2\\x_3\\x_4\end{bmatrix} =\boldsymbol{x}.

Thus the addition site makes no numerical change. Calling it “no meaningful change” additionally assumes that the state and zero update use the same four-coordinate representation and that no other operation—such as normalization—changes the value afterward.

Question 5

Can [1, 2]ᵀ + [3, 4, 5]ᵀ be a vector sum? Diagnose the problem, then state one architecture-level remedy for a residual block with mismatched dimensions.

Show the step-by-step solution

The first vector belongs to $\mathbb{R}^{2}$ and the second to $\mathbb{R}^{3}$ . Componentwise addition would need a matching second input for each output coordinate, but the first vector has no third coordinate. Therefore the vector sum is undefined as written.

A residual architecture can first map one branch into the other’s dimension. For example, a projection $P$ could produce $P\boldsymbol{x}\in\mathbb{R}^{3}$ from $\boldsymbol{x}\in\mathbb{R}^{2}$ . The defined addition would then be $P\boldsymbol{x}+\boldsymbol{u}$ , provided the projection also establishes the intended coordinate semantics. Padding is another architecture-specific choice, but silently dropping or inventing a coordinate is not vector addition.

Question 6

Two shape-(3,) arrays use feature orders (clicks, purchases, returns) and (returns, clicks, purchases). Why is adding them position by position invalid, and how should the second array be prepared?

Show the step-by-step solution

Shape equality pairs position 1 with position 1, but the labels show that this would add clicks to returns. Positions 2 and 3 are also misaligned. The operation would produce numbers without a consistent feature interpretation.

First attach each value in the second array to its label. Then reorder those labeled values into

(\texttt{clicks},\texttt{purchases},\texttt{returns}),

the convention used by the first array. Only after both arrays agree on dimension, order, and units should positionwise addition run. NumPy cannot perform this semantic check from bare numeric arrays because their feature labels are absent.

Question 7

A batch has shape (4, 3), and update has shape (3,). NumPy accepts batch + update. What output shape results, what calculation occurs, and which question must the programmer answer before trusting it?

Show the step-by-step solution

NumPy compares trailing dimensions. The final dimensions are both 3, while the missing leading dimension of update acts like 1 and can expand across the batch’s four rows. The output therefore has shape (4, 3).

For each row index $i\in\{0,1,2,3\}$ and coordinate $j\in\{0,1,2\}$ , NumPy computes

\texttt{result}[i,j]=\texttt{batch}[i,j]+\texttt{update}[j].

The program applies the same three-coordinate update to all four rows. The programmer must decide whether a shared update was intended. Successful broadcasting proves shape compatibility under NumPy’s rules; it does not prove that the model contract intended one update to be reused across every item.

Question 8

A sublayer receives x = [1, −2, 3]ᵀ and returns F(x) = [0.5, 2, −1]ᵀ. Compute the residual sum, then explain precisely what the arithmetic does and does not establish.

Show the step-by-step solution

Add matching coordinates:

\begin{aligned} \boldsymbol{y} &=\boldsymbol{x}+\mathcal{F}(\boldsymbol{x})\\ &= \begin{bmatrix} 1+0.5\\ -2+2\\ 3+(-1) \end{bmatrix}\\ &= \begin{bmatrix}1.5\\0\\2\end{bmatrix}. \end{aligned}

The arithmetic establishes that the compatible input and sublayer output were combined coordinatewise into one three-coordinate result. It does not identify the human meaning of any coordinate, prove that the original information is fully recoverable, or show what a later normalization or layer will do. Those are properties of the learned representation and surrounding architecture, not consequences of the plus sign alone.

One coordinate rule combines the whole vector

Follow a residual update without skipping a coordinate

The diagonal and the two-step path reach the same point

Three algebraic facts follow coordinate by coordinate

Residual connections use this exact addition mechanism

NumPy adds values and broadcasts shapes

A successful array operation can still be the wrong addition

Check your understanding

Sources

Follow new Math for AI articles

Keep reading

Subspaces for AI: Test Restricted Feature Families

Linear Equations for AI: Turn One Formula Into a Geometric Line

Linear Algebra in a Tiny Neural Network