How Affine Spaces Explain Bias and Decision Boundaries

A classifier’s weights decide which direction separates two classes. Its bias lets the separating line sit somewhere other than the origin. That small addition is a geometric translation: it changes a linear boundary into an affine one while keeping the boundary’s direction fixed.

By the end, you will be able to distinguish a translated subspace from a subspace through the origin, test whether a mapping is linear or affine, and calculate how a bias changes a simple classifier’s boundary.

Learning outcome: Distinguish linear and affine sets and transformations.

This builds on Lesson 034, which showed that linear mappings preserve vector sums and scaling. Section 2.8 of Mathematics for Machine Learning extends that structure with offsets. The examples, classifier calculation, visual, and assessments here are original.

A line can keep its direction and move away from zero

A vector subspace must contain the zero vector. For example, all multiples of $d=(1,2)$ form a line through the origin:

U = \{t(1,2):t\in\mathbb{R}\}.

Choose a point $p=(0,-1)$ and add it to every vector in $U$ . The resulting set is a parallel line that no longer passes through the origin:

L = p+U = \{p+u:u\in U\} = \{(0,-1)+t(1,2):t\in\mathbb{R}\}.

For this line, the parametric coordinates are $(x_1,x_2)=(t,2t-1)$ . Eliminating $t$ gives $2x_1-x_2=1$ . The direction vector is unchanged from $U$ , but every point on $L$ has moved by $p$ . In particular, $0\notin L$ : making the first coordinate zero forces $t=0$ , which leaves the second coordinate at $-1$ . When the chosen point $p$ is in $U$ , the translate $p+U$ equals $U$ ; that is the special case where the affine subspace is also a vector subspace.

The same direction, with and without an offsetThe bias changes the boundary from 2x₁ − x₂ = 0 to 2x₁ − x₂ = 1. The lines stay parallel, and the translated boundary no longer contains the origin.

No bias: 2x₁ − x₂ = 0
Bias −1: 2x₁ − x₂ = 1

The two lines in the figure have the same direction vector $(1,2)$ , since it is perpendicular to their shared normal vector $(2,-1)$ . The dashed line is a translation of the solid one. Its position changes, but its orientation does not. The labels and equations give the same information without relying on the colors.

Affine combinations preserve points in an offset set

A useful way to recognize affine geometry is to combine points with coefficients that sum to one. If $x=p+u$ and $y=p+v$ are in $L$ , then for any real number $\lambda$ ,

\lambda x+(1-\lambda)y = \lambda(p+u)+(1-\lambda)(p+v) = p+\bigl(\lambda u+(1-\lambda)v\bigr) \in p+U.

The offset $p$ remains because $\lambda+(1-\lambda)=1$ , and the part in parentheses remains in $U$ because subspaces are closed under linear combinations. This is why points on a line stay on that line under interpolation, including extrapolation when $\lambda$ is outside $[0,1]$ .

Ordinary sums have a different coefficient total. For $x,y\in L$ , $x+y=2p+(u+v)$ , which need not be in $p+U$ . With the example line, $x=p$ and $y=p$ give $x+y=(0,-2)$ , which fails the line equation $2x_1-x_2=1$ . So an affine subspace keeps affine combinations, while a vector subspace keeps all linear combinations and includes zero.

A chosen anchor is not unique. If $q\in U$ , then $p+U=(p+q)+U$ because adding an element of $U$ only changes which point is used as the anchor. The direction space is the invariant part; the support point is one convenient representative.

A bias moves a classifier boundary

An affine mapping combines a linear map with a fixed translation:

For any two inputs, the translation cancels from their difference:

F(x)-F(y)=(Ax+b)-(Ay+b)=A(x-y).

And for coefficients that add to one, the output respects their affine combination:

F\bigl(\lambda x+(1-\lambda)y\bigr) = \lambda F(x)+(1-\lambda)F(y).

This differs from the linear-map rule for arbitrary sums. In particular, $F(0)=b$ , so a nonzero bias moves the zero input away from the zero output.

Let $x=(x_1,x_2)\in\mathbb{R}^2$ be a two-feature input. Its score is

s(x)=2x_1-x_2-1.

It assigns the positive class when $s(x)\ge 0$ and the negative class when $s(x)<0$ . The decision boundary is where the score is zero:

2x_1-x_2-1=0 \quad\Longleftrightarrow\quad 2x_1-x_2=1.

The weight vector $w=(2,-1)$ sets the boundary’s normal, while the bias $b=-1$ sets the score level at $w^\mathsf{T}x=1$ . In two dimensions the boundary is a line; in higher-dimensional feature spaces, the corresponding flat set is called a hyperplane. The point $p=(0,-1)$ is on it because $2(0)-(-1)-1=0$ . Adding the direction $d=(1,2) does not change the score:

Without the bias, the boundary would be $2x_1-x_2=0$ and would contain the origin. Adding $b$ changes its intercept, not its normal vector. More broadly, a classifier score $w^\mathsf{T}x+b$ uses $w$ to orient a hyperplane and $b$ to set its offset. PyTorch’s torch.nn.Linear documents this affine form as $y=xA^\mathsf{T}+b$ and enables its learnable bias by default (torch.nn.Linear documentation). With one output, the layer can produce a score of this form; a following rule such as thresholding that score assigns the class.

The bias moves a boundary parallel to itself only while the weights stay fixed. If training changes $w$ , it changes the orientation too. A bias also cannot make one affine boundary represent every possible classification pattern; it only adjusts the offset of this particular linear score.

Verify the geometry with NumPy

The example can be checked directly. The code computes the scores at the support point and two points reached by moving along the direction:

import numpy as np

w = np.array([2.0, -1.0])
b = -1.0
p = np.array([0.0, -1.0])
d = np.array([1.0, 2.0])
score = lambda x: w @ x + b

print(score(p))          # 0.0: p is on the boundary
print(score(p + d))      # 0.0: moving along d stays on it
print(score(p + 3 * d))  # 0.0: extrapolating along d also stays on it
print(score(np.zeros(2)))  # -1.0: the origin is off the boundary

The calculations use column-vector geometry, while common tensor batches store examples in rows. PyTorch writes its batched linear layer as $xA^\mathsf{T}+b$ ; the transpose accounts for that row-oriented convention. With its default bias, the operation is affine. Setting bias=False makes the layer’s mapping linear, though a later nonlinear activation is a separate operation.

Check your understanding

Question 1

For U = span((1, 2)) and p = (0, −1), write p + U parametrically and state whether it contains the origin.