Linear Combinations for AI: How Weighted Vectors Build a New Representation

An AI system often has several candidate vectors and needs one result. Picking only one candidate would discard the rest. Adding them unchanged would give every candidate the same strength. A linear combination supplies a more flexible operation: scale each vector by its own coefficient, then add the scaled contributions.

For example, two vectors might encode different pieces of context. Coefficients of $0.75$ and $0.25$ let the first contribute three times as strongly as the second. The result is still one vector in the same coordinate system, ready for the next computation.

By the end of this lesson, you will be able to construct a linear combination coordinate by coordinate, interpret what each coefficient changes, implement the operation with shape checks in NumPy, and identify the extra conditions that turn a general linear combination into an average-like attention output.

Learning outcome: Construct and interpret a weighted sum of vectors.

Lesson 16: Scalar Multiplication for AI showed how one scalar controls one vector’s strength and direction. Lesson 15 supplied vector addition. This lesson joins those operations. The official Mathematics for Machine Learning companion places scalar and matrix operations in Section 2.2 and defines linear combinations in Section 2.5 of the book. The feature story, calculations, code, visual, and assessments below are original.

Give every candidate its own coefficient

Let $\boldsymbol{v}_1,\boldsymbol{v}_2,\ldots,\boldsymbol{v}_k\in\mathbb{R}^n$ be $k$ vectors. The notation $\mathbb{R}^n$ means that each vector has $n$ real coordinates. Let $a_1,a_2,\ldots,a_k\in\mathbb{R}$ be real-number coefficients. A coefficient is the scalar assigned to one vector.

Calling this a “weighted sum” describes the same arithmetic: the coefficients are the weights. “Weight” here does not necessarily mean a learned model parameter or a probability. Its role is simply to scale one contribution.

All input vectors must belong to the same vector space for their sum to make sense. In an array calculation, that normally means the same shape and the same coordinate meaning. A two-coordinate vector for [temperature, humidity] cannot be added meaningfully to a two-coordinate vector for [height, age] merely because both have shape (2,).

Track two contributions all the way to the result

Suppose a small representation uses the coordinates [topic, urgency]. Two candidate vectors are

Choose coefficients $a=0.75$ and $b=0.25$. The requested linear combination is $\boldsymbol{r}=a\boldsymbol{u}+b\boldsymbol{v}$.

This example also shows why “larger coefficient” is not the same as “larger final coordinate.” A coefficient scales a whole vector. The sign and size of each vector coordinate still determine whether that contribution raises or lowers the corresponding result coordinate.

The component rule explains the operation

Write the input vectors by coordinates:

The symbol $v_{ij}$ means coordinate $j$ of vector $i$. Scaling and addition act coordinate by coordinate, so the result’s coordinate $j$ is

This identity is the reason the two-step recipe works. Scalar multiplication makes contribution $i$ equal to $[a_i v_{i1},\ldots,a_i v_{in}]^\mathsf{T}$. Vector addition then sums all first coordinates to obtain $r_1$, all second coordinates to obtain $r_2$, and so on. Because each coordinate calculation is a real-number calculation, the result again has $n$ real coordinates.

The same rule gives useful algebraic checks. Regrouping contributions does not change the result because real-number addition is associative. Reordering them does not change it because addition is commutative:

That does not mean coefficients can be swapped between vectors. Usually $a\boldsymbol{u}+b\boldsymbol{v}\ne b\boldsymbol{u}+a\boldsymbol{v}$ because a different vector received each scale.

Move the coefficients, not the coordinate meanings

The explorer fixes $\boldsymbol{u}=[3,1]^\mathsf{T}$ and $\boldsymbol{v}=[-1,3]^\mathsf{T}$ while exposing $a$ and $b$. The solid and dashed arrows show the two scaled contributions from the origin. A translated copy of $b\boldsymbol{v}$ begins at the tip of $a\boldsymbol{u}$, showing the addition path. The dotted result arrow connects the origin directly to their sum.

At the server-rendered starting state, $a=0.75$ and $b=0.25$, so the dotted arrow ends at $(2,1.5)$, matching the worked example. Set one coefficient to zero and its contribution disappears. Move a coefficient below zero and its arrow reverses. Set both coefficients to zero and every contribution collapses to the zero vector. None of those changes renames or reorders the coordinates.

A linear combination is broader than an average

Several related terms use the same scale-and-add operation but impose different rules on the coefficients.

In this lesson, weighted average means the common form with normalized nonnegative weights, so it is a convex combination. Some fields use the term “weighted mean” with different conventions, which should be stated explicitly. The starting example qualifies under the convention here because $0.75+0.25=1$ and both coefficients are nonnegative. In contrast, $1.5\boldsymbol{u}-0.5\boldsymbol{v}$ is a valid linear and affine combination, but not a convex combination: its negative coefficient can carry the result beyond the segment between the two vectors.

The zero vector is always reachable as a linear combination by setting every coefficient to zero. That fact does not say the inputs are themselves zero. It only says the operation permits the trivial combination with no active contribution.

Attention uses a constrained weighted sum of values

The original Transformer paper defines an attention output as a weighted sum of value vectors. Each value vector is a candidate representation with the same feature dimension. The attention mechanism computes how much each value contributes for the current position, then performs the linear combination. The paper’s scaled dot-product attention equation writes many such sums compactly as a matrix operation.

For a single illustrative output, suppose the already-computed attention weights are $0.6$, $0.3$, and $0.1$, and the value vectors are

Then the attention-style representation is

In the paper’s attention operation, the normalization used to produce these weights makes them nonnegative and makes each set sum to one. The final sum is therefore a convex combination of the value vectors for that attention output. Those restrictions belong to this attention mechanism, not to the definition of a linear combination.

This lesson deliberately starts after the weights have been produced. It does not yet teach queries, keys, compatibility scores, normalization, multiple heads, or the later output projection. The durable mathematical point is narrower: once coefficients and compatible value vectors are available, attention builds a representation by scaling each value and adding the contributions.

NumPy should reveal the vector axis

Store the two candidate vectors as rows of one array. The array vectors then has shape (2, 2): two vectors, each with two coordinates. The coefficient array has shape (2,), one coefficient per vector.

import numpy as np

vectors = np.array([
    [3.0, 1.0],   # u
    [-1.0, 3.0],  # v
])
coefficients = np.array([0.75, 0.25])

if vectors.ndim != 2:
    raise ValueError("vectors must have shape (number_of_vectors, coordinates)")
if coefficients.shape != (vectors.shape[0],):
    raise ValueError("need exactly one coefficient per vector")

scaled = coefficients[:, np.newaxis] * vectors
result = scaled.sum(axis=0)

print(scaled)
print(result)

The verified output is

[[ 2.25  0.75]
 [-0.25  0.75]]
[2.  1.5]

coefficients[:, np.newaxis] changes the coefficient shape from (2,) to (2, 1). NumPy can then broadcast one coefficient down each vector row. The official broadcasting rules compare array dimensions from the right and allow dimensions that match or have size one. After multiplication, sum(axis=0) sums down the vector-row axis and preserves the two coordinate columns.

Writing only coefficients * vectors fails for this example: trailing shapes (2,) and (2, 2) happen to align the two coefficients with the coordinate columns, so NumPy returns a plausible-looking array with the wrong meaning. The first row becomes [2.25, 0.25] instead of [2.25, 0.75]. A successful broadcast is not proof that the coefficient axis was correct.

Shape compatibility is necessary but still not sufficient. If one candidate has three coordinates and another has two, there is no single rectangular vectors array of ordinary numeric values to reduce this way. If both have two coordinates but use different feature definitions, the code runs while the result remains semantically invalid. Check both array shape and coordinate meaning.

Coefficients answer “how much,” not “what does this mean?”

Three habits prevent most errors:

1. Pair every coefficient with a specific vector before calculating.
2. Scale complete vectors, then add matching coordinates.
3. State any extra coefficient rule—such as nonnegativity or a sum of one—rather than assuming the phrase “weighted sum” supplies it.

Negative coefficients are not mistakes in a general linear combination. They reverse a contribution. Coefficients larger than one are also valid; they can extend the result beyond the input arrows. Cancellation is possible when contributions point against one another, and a zero coefficient removes a candidate without changing the vector’s shape.

The next prerequisite-safe course step, Lesson 18, organizes numbers into matrices. Lesson 19 will then reveal a matrix–vector product as several weighted sums. Keep the current recipe—pair, scale, add—close at hand; matrix notation will compress it, not replace it. The Math for AI course page keeps the ordered path and published prerequisites together.

Check your understanding

scaled = coefficients[:, np.newaxis] * vectors
result = scaled.sum(axis=0)