Dimension Counts Independent Directions

An embedding batch can have four rows and hundreds of coordinates per row. The first number tells you how many inputs were encoded; the second tells you the coordinate width of each output. Neither number, by itself, tells you how many independent directions the observed vectors actually use.

In this lesson, we will read those counts separately, connect dimension to the number of vectors in a basis, and see why two arrays with the same shape can span spaces of different dimensions. The distinction matters when reading an embedding API, estimating what a batch can tell you, or describing a representation without confusing stored examples with the space that holds them.

The previous lesson showed how a basis gives every vector a coordinate description. Lesson 31: Bases give compact coordinate systems established the key idea we need here: a basis spans a space without redundant directions. The official Mathematics for Machine Learning companion site develops dimension in Section 2.6 of the book PDF. The small datasets, code, visual, and assessments in this lesson are original.

A basis gives dimension its count

Recall that a basis is a linearly independent set of vectors that spans a vector space. If every vector in a space $V$ can be described using a basis $B$ , then the dimension of $V$ is the number of vectors in that basis:

\dim(V)=|B|.

The notation $|B|$ means “the number of elements in the set $B$ .” It does not count the number of entries used to write any one basis vector. Every basis of a finite-dimensional space has the same number of vectors, even though the actual basis vectors can differ. That count is what makes dimension a property of the space rather than of one particular coordinate system.

The vectors above each contain three coordinates, but that is not the definition of dimension. The decisive fact is that the three vectors form a basis: together they generate every vector in $\mathbb{R}^3$ , and none is a linear combination of the other two. A different basis can point in other directions and still contain exactly three vectors.

Four samples can still live in three dimensions

Suppose an encoder returns four vectors in $\mathbb{R}^3$ :

x_1=e_1,\qquad x_2=e_2,\qquad x_3=e_3,\qquad x_4=e_1+e_2+e_3.

Store each vector as one row of a matrix:

X_{\text{full}}= \begin{bmatrix} 1&0&0\\ 0&1&0\\ 0&0&1\\ 1&1&1 \end{bmatrix}\in\mathbb{R}^{4\times3}.

Its shape is $(4,3)$ : four rows and three columns. Here, rows are samples and columns hold the three coordinates of each sample. The first three rows are the standard basis, so the four samples span all of $\mathbb{R}^3$ . The fourth row adds another sample, but not another independent direction, because $x_4=x_1+x_2+x_3$ . The vector space has dimension three; the batch has four examples.

Now compare this batch:

X_{\text{line}}= \begin{bmatrix} 1&0&0\\ 2&0&0\\ -1&0&0\\ 3&0&0 \end{bmatrix}\in\mathbb{R}^{4\times3}.

It has the same shape, $(4,3)$ , yet every row is a multiple of $e_1=(1,0,0)$ . One vector, $e_1$ , spans all four samples, so their span is a line with dimension one. The ambient coordinate space is still $\mathbb{R}^3$ ; the particular samples occupy only a one-dimensional subspace of it.

Embedding dimension and batch shape

The same four-by-three shape can span three directions or one

Both batches contain four samples with three coordinates each. The coordinate tables show the exact values; the diagrams summarize how many independent directions their rows use.

Three independent directions

Shape: 4 samples × 3 coordinates

The first three rows are the standard basis of three-dimensional space. The fourth is their sum, so the sample span has dimension three.
Sample	Coordinate 1	Coordinate 2	Coordinate 3
x₁	1	0	0
x₂	0	1	0
x₃	0	0	1
x₄	1	1	1

The first three rows are the standard basis of three-dimensional space. The fourth is their sum, so the sample span has dimension three.

Dimension of the sample span3 directions

One independent direction

Shape: 4 samples × 3 coordinates

Each row is a multiple of the first coordinate direction. Four samples therefore span only a line, whose dimension is one.
Sample	Coordinate 1	Coordinate 2	Coordinate 3
x₁	1	0	0
x₂	2	0	0
x₃	−1	0	0
x₄	3	0	0

Each row is a multiple of the first coordinate direction. Four samples therefore span only a line, whose dimension is one.

Dimension of the sample span1 direction

Rows count inputs, columns count stored coordinates, and basis vectors count independent directions. The table is the exact data; the small direction sketches are schematic.

A shape reports how an array is stored. It does not reveal whether its rows are independent; the values and their relationships determine the dimension of their span.

The diagram keeps three questions separate: how many inputs were encoded, how many coordinates each output contains, and how many directions those particular outputs span. The arrays share the same shape, but only the first batch reaches every direction in $\mathbb{R}^3$ .

NumPy reports shape, not a basis

NumPy’s ndarray.shape is a tuple containing the length of each array axis. For a two-dimensional matrix stored as $N$ rows of $d$ coordinates, that tuple is (N, d). The shape tells us how the values are arranged. To identify the dimension of a span, we must also reason about whether some rows can be built from others.

Here is the synthetic example encoded directly in NumPy:

import numpy as np

full = np.array([
    [1.0, 0.0, 0.0],
    [0.0, 1.0, 0.0],
    [0.0, 0.0, 1.0],
    [1.0, 1.0, 1.0],
])

line = np.array([
    [1.0, 0.0, 0.0],
    [2.0, 0.0, 0.0],
    [-1.0, 0.0, 0.0],
    [3.0, 0.0, 0.0],
])

print(full.shape, line.shape)
print(np.allclose(full[3], full[0] + full[1] + full[2]))
print(np.allclose(
    line,
    np.array([1.0, 2.0, -1.0, 3.0])[:, None]
    * np.array([1.0, 0.0, 0.0]),
))

The output is:

(4, 3) (4, 3)
True
True

The first line confirms that both matrices have four rows and three coordinates per row. The first True verifies that the last full sample is the sum of the three basis rows. The second verifies that each line sample is the stated scalar multiple of $e_1$ . Those relationships justify the span dimensions: three for full, one for line.

The NumPy shape documentation defines shape as the tuple of array dimensions. That is a storage description, not a promise that all coordinates vary independently. A quick check of len(full.shape) would return two because full has two array axes. That number is also distinct from the dimension three of its embedding space.

The batch axis is not the embedding axis

The distinction appears in real AI code as well. The Sentence Transformers usage documentation encodes three sentences with all-MiniLM-L6-v2 and reports an embedding array with shape [3, 384]. The first length corresponds to the three supplied sentences; the second is the output coordinate width. The model API exposes get_embedding_dimension() for the output width, and its encode method returns embeddings for the provided inputs.

For a million stored documents with the same encoder, the corpus may contain a million rows, while each row remains 384 numbers wide. Increasing the corpus adds more vectors; it does not turn the embedding space into a million dimensions. Conversely, four outputs in $\mathbb{R}^{384}$ cannot span all 384 dimensions, because four vectors supply at most four independent directions. They might span fewer if some outputs repeat a direction or are linear combinations of others.

One more boundary matters: the output width describes the chosen model and configuration, not a universal property of the texts. Sentence Transformers allows an output truncation dimension in encode; after such a configuration change, the returned coordinate width changes too. The encode API documents the truncate_dim option. The vectors still need to be compared in a compatible coordinate space. And a larger dimension alone does not guarantee that an embedding preserves more useful meaning; that requires evaluating the representation for its intended task.

Dimension gives us a count of independent basis directions. A shape gives us the lengths of array axes, and a batch gives us a collection of samples. Once those quantities stop sharing one label, the next question becomes easier to ask: how many independent directions do the actual rows contain? Lesson 33 introduces matrix rank to count that effective dimension systematically.

Check your understanding

Question 1

A basis B for a vector space contains five vectors. What is the dimension of the space, and why does the number of coordinates printed inside one basis vector not define it?