Powers for AI: Why Doubling Context Can Quadruple Attention Work

A sequence of $2{,}048$ tokens contains $4{,}194{,}304$ ordered token pairs. Double the sequence to $4{,}096$ tokens and the pair count becomes $16{,}777{,}216$—four times as many. Nothing about the number $2$ changed. The operation applied to it did.

That operation is a power. Powers compress repeated multiplication into a small superscript, and the superscript tells us how a quantity responds when its input changes scale. By the end of this lesson, you will be able to manipulate positive, zero, negative, and fractional exponents; treat roots as inverse powers with real-number domain limits; and calculate when an AI quantity grows linearly or quadratically.

This lesson builds on Lesson 2’s view of functions as reusable rules. Here, the rule may be $f(n)=n^2$, and the question is how its output changes when $n$ changes. The official Mathematics for Machine Learning companion separates mathematical foundations from the machine-learning systems that use them. Its introductory chapter motivates those foundations as tools for understanding models and their assumptions; the exponent algebra below is an original prerequisite bridge developed for this course.

One superscript controls the growth rule

In $a^p$, $a$ is the base and $p$ is the exponent. The exponent says how the base participates in the operation.

The intuition is “multiply the base by itself $p$ times,” but that only defines positive whole-number exponents. The formal starting point is:

Zero and negative exponents extend this definition while preserving the same algebraic laws. For any nonzero $a$,

$$a^0=1 \qquad\text{and}\qquad a^{-p}=\frac{1}{a^p}.$$

The condition $a\ne0$ matters. A negative power creates a reciprocal, and division by zero is undefined. For example,

$$2^{-3}=\frac{1}{2^3}=\frac{1}{8},$$

whereas $0^{-3}$ has no real value.

The zero-exponent rule follows from the quotient law rather than from “no multiplication at all.” If $a\ne0$, then

$$\frac{a^3}{a^3}=1 \qquad\text{and}\qquad \frac{a^3}{a^3}=a^{3-3}=a^0.$$

Both expressions can agree only if $a^0=1$. Because the cancellation used $a\ne0$, this argument does not decide $0^0$; that boundary needs its own context.

Exponent laws account for every factor

For positive integers $m$ and $n$, multiplying two powers with the same base joins their repeated factors:

$$a^m a^n =\underbrace{a\cdots a}_{m\text{ factors}} \underbrace{a\cdots a}_{n\text{ factors}} =a^{m+n}.$$

The same counting idea gives the core laws below. Negative exponents require nonzero bases, and fractional exponents require the domain care introduced in the next section.

Operation	Law	Necessary caution
Multiply like bases	$a^m a^n=a^{m+n}$	Both powers must be defined
Divide like bases	$\dfrac{a^m}{a^n}=a^{m-n}$	$a\ne0$
Raise a power	$(a^m)^n=a^{mn}$	Check the real-number domain for fractional powers
Raise a product	$(ab)^n=a^n b^n$	Always safe for integer $n$; use care with real exponents

One tempting pattern is not an exponent law:

$$(a+b)^p\ne a^p+b^p\quad\text{in general}.$$

For $a=2$, $b=3$, and $p=2$, the left side is $5^2=25$, while the proposed right side is $2^2+3^2=13$. Addition inside parentheses must be handled before the power, or expanded with the distributive law.

Roots reverse powers, with a domain attached

The principal square root of a nonnegative real number $y$, written $\sqrt{y}$, is the nonnegative real number $r$ whose square is $y$:

$$r=\sqrt{y} \quad\Longleftrightarrow\quad r\ge0\text{ and }r^2=y.$$

Thus $\sqrt{49}=7$. Although both $7^2$ and $(-7)^2$ equal $49$, the radical symbol $\sqrt{49}$ names the nonnegative root. Solving the equation $x^2=49$ is a different task and has two real solutions, $x=7$ and $x=-7$.

For $a\ge0$, the square root is the one-half power:

$$a^{1/2}=\sqrt{a}.$$

More generally, for positive $a$ and positive integer $q$, $a^{1/q}=\sqrt[q]{a}$. A rational exponent combines a root and a power:

$$a^{m/q}=\left(\sqrt[q]{a}\right)^m.$$

A square root does not mean “divide by two.” The number $36/2$ is $18$, while $\sqrt{36}=6$ because $6^2=36$. The exponent is what is halved: $\sqrt{a^4}=(a^4)^{1/2}=a^2$ for real $a$. More generally, $\sqrt{x^2}=|x|$, not always $x$; for $x=-5$, both sides equal $5$.

A doubling test exposes linear and quadratic growth

Suppose a quantity follows the power rule

$$f(n)=C n^p,$$

where $n>0$ is an input size, $p$ is the scaling exponent, and $C>0$ is a constant that does not change during the comparison. If the input is multiplied by a scale factor $k>0$, then

$$f(kn)=C(kn)^p=Ck^pn^p=k^p f(n).$$

The output therefore changes by the factor $k^p$. With a doubling, $k=2$:

Exponent $p$	Rule	Effect of $n\to2n$	Name used here
$1$	$Cn$	$2^1=2$ times	linear growth
$2$	$Cn^2$	$2^2=4$ times	quadratic growth
$3$	$Cn^3$	$2^3=8$ times	cubic growth
$-1$	$Cn^{-1}=C/n$	$2^{-1}=1/2$ as large	inverse growth

The explorer places powers of two at equal distances on each axis. That is a logarithmic scale: equal steps represent equal multiplication factors rather than equal additions. Move the sequence-length control with a pointer or the arrow keys and compare the straight solid line, $n$, with the dashed square curve, $n^2$.

Sequence length n

Choose a power-of-two length from 128 to 8,192 tokens. Arrow keys change one doubling at a time.

1,024

Linear nQuadratic n²

Selected length

1,024

Linear growth

8×

Quadratic growth

64×

n² score positions

1,048,576

At n = 1,024, linear growth is 8× the baseline and quadratic growth is 64×, giving 1,048,576 score positions.

Show the chart data as a table

Linear and quadratic growth relative to 128 tokens

Sequence length n	Linear n	Quadratic n²	Score positions n²	Current state
128	1×	1×	16,384
256	2×	4×	65,536
512	4×	16×	262,144
1,024	8×	64×	1,048,576	← Selected
2,048	16×	256×	4,194,304
4,096	32×	1,024×	16,777,216
8,192	64×	4,096×	67,108,864

The server-rendered starting state selects $n=1{,}024$. Relative to $128$, the length is $8$ times as large, but the square is $8^2=64$ times as large. The table beneath the chart keeps every plotted value available without relying on the visual shape or on JavaScript.

Dense attention turns token pairs into a square

The original Transformer paper defines scaled dot-product attention with $QK^{\mathsf T}$: each query is compared with every key before the scores are scaled, normalized, and used to combine values. In its classical dense self-attention analysis, the paper reports a per-layer complexity term of $O(n^2d)$, where $n$ is sequence length and $d$ is representation width (Vaswani et al., Attention Is All You Need).

The matrix notation can be read as a grid even before the course formally introduces matrices. With $n$ token positions:

• there are $n$ query rows;
• each row has one score position for each of $n$ keys; and
• the full score grid therefore contains $n\times n=n^2$ positions per head.

If each query-key comparison uses $d_k$ features, forming those dot products has a leading arithmetic term proportional to $n^2d_k$. Holding $d_k$ fixed while changing only $n$ gives the doubling calculation from the opening:

This count is a mathematical property of the full dense score grid. It does not guarantee that wall-clock time or peak memory will rise by exactly $4$ times. Parallel hardware, tiling, fused or recomputed kernels, masking, and attention mechanisms that avoid the full grid can change what an implementation stores and how quickly it runs. The safe claim is narrower: classical dense self-attention has $n^2$ query-key score positions, and its standard arithmetic complexity contains a quadratic sequence-length term.

Parameter growth depends on which dimensions change

A dense layer that maps $d_{\text{in}}$ input features to $d_{\text{out}}$ output features needs one weight for every input-output pair:

$$N_{\text{weights}}=d_{\text{in}}d_{\text{out}}.$$

An optional bias adds $d_{\text{out}}$ more parameters, but the pairwise weight table is the main scaling relationship in this example.

Suppose the layer changes from $512$ inputs and $2{,}048$ outputs to $1{,}024$ inputs and $4{,}096$ outputs. Both dimensions doubled:

$$\begin{aligned} N_{\text{old}} &= 512\times2{,}048=1{,}048{,}576,\\ N_{\text{new}} &= 1{,}024\times4{,}096=4{,}194{,}304. \end{aligned}$$

The ratio makes the exponent visible:

$$\frac{N_{\text{new}}}{N_{\text{old}}} =\frac{(2\cdot512)(2\cdot2{,}048)}{512\cdot2{,}048} =2\cdot2 =2^2 =4.$$

If only the output width doubled, the count would double instead. Calling all parameter growth “quadratic” hides which dimensions actually changed. A square law appears here only when two multiplied dimensions scale together.

NumPy lets the formula and its inverse check each other

NumPy’s np.logspace constructs values at equal intervals on a log scale; with base $2$, integer endpoints $7$ through $13$ generate the context lengths $2^7$ through $2^{13}$. Its np.power raises corresponding array elements to powers, and np.sqrt returns the nonnegative square root element by element.

import numpy as np

lengths = np.logspace(7, 13, num=7, base=2, dtype=np.int64)
score_positions = np.power(lengths, 2)
quadratic_growth = score_positions // score_positions[0]
recovered_lengths = np.sqrt(score_positions)

print("lengths:", lengths)
print("score positions:", score_positions)
print("quadratic growth:", quadratic_growth)
print("recovered lengths:", recovered_lengths)

lengths: [ 128  256  512 1024 2048 4096 8192]
score positions: [   16384    65536   262144  1048576  4194304 16777216 67108864]
quadratic growth: [   1    4   16   64  256 1024 4096]
recovered lengths: [ 128.  256.  512. 1024. 2048. 4096. 8192.]

The final line checks the inverse relationship on nonnegative inputs: $\sqrt{n^2}=n$ because every sequence length in the array is positive. For an array that could contain negative $n$, the correct identity would be $\sqrt{n^2}=|n|$.

The boundary cases are part of the operation

Exponent notation is compact enough to hide a domain error. These cases should remain visible when reading equations or code:

• $0^0$ depends on context. The rule $a^0=1$ above assumed $a\ne0$, while $0^p=0$ for positive $p$. At $0^0$, those extensions meet without deciding one value. Some combinatorial formulas and software systems define it as $1$ for convenience; elementary real exponent algebra often leaves it undefined. State the convention instead of silently choosing one.
• Even roots of negative real numbers are not real. No real $r$ satisfies $r^2=-9$, so $\sqrt{-9}$ has no real value. Complex numbers extend the domain, but they are outside this lesson.
• A negative base with a fractional exponent needs special care. The real cube root $\sqrt[3]{-8}=-2$ exists, so the exact rational expression $(-8)^{1/3}$ can be interpreted as $-2$. A floating-point exponent such as 1 / 3 is an approximation, however, and NumPy’s real-valued np.power returns nan for a negative base with a non-integral exponent. Do not assume software will reconstruct the intended fraction.
• A power does not distribute over addition. The counterexample $(2+3)^2=25\ne13=2^2+3^2$ is enough to reject that shortcut.
• A root does not divide the value by its index. $\sqrt{36}=6$, not $18$. Roots divide exponents when the relevant domain assumptions hold.

The log-scale explorer already hints at the next question. If $2^p=8$, which operation recovers $p=3$? Lesson 8 introduces logarithms as inverse powers and uses them to turn multiplicative scale into additive steps. Until it is live, the Math for AI course page keeps the verified publication order.

Check your understanding

Question 1

Evaluate 3⁴ by expanding the power, and identify the base and exponent.

Show the step-by-step solution

In $3^4$, the base is $3$ and the exponent is $4$. A positive integer exponent counts repeated factors of the base:

$$3^4=3\times3\times3\times3.$$

Multiply in stages:

$$3\times3=9, \qquad 9\times3=27, \qquad 27\times3=81.$$

Therefore $3^4=81$. Multiplying $3\times4=12$ would confuse a factor count with ordinary multiplication.

Question 2

For x ≠ 0, simplify x³x⁻⁵, rewrite the result without a negative exponent, and check it at x = 2.

Show the step-by-step solution

The bases match, so multiplication adds the exponents:

$$x^3x^{-5}=x^{3+(-5)}=x^{-2}.$$

A negative exponent means reciprocal, and the assumption $x\ne0$ makes that reciprocal valid:

$$x^{-2}=\frac{1}{x^2}.$$

At $x=2$,

$$2^3\cdot2^{-5} =8\cdot\frac{1}{32} =\frac{1}{4}.$$

The simplified form gives $1/2^2=1/4$ as well, so the calculation checks.

Question 3

Evaluate 64⁻²ᐟ³ and explain the role of the negative sign, numerator 2, and denominator 3 in the exponent.

Show the step-by-step solution

The negative sign asks for a reciprocal:

$$64^{-2/3}=\frac{1}{64^{2/3}}.$$

The denominator $3$ asks for a cube root, and the numerator $2$ then asks for a square:

$$64^{2/3}=\left(\sqrt[3]{64}\right)^2=4^2=16.$$

Combining the steps gives

$$64^{-2/3}=\frac{1}{16}.$$

The base $64$ is positive and nonzero, so both the root and reciprocal are valid in the real numbers.

Question 4

A dense layer grows from 300 × 200 weights to 600 × 400 weights. Calculate both counts and explain why doubling both dimensions produces a fourfold change.

Show the step-by-step solution

The original layer has one weight for each input-output pair:

$$300\times200=60{,}000\text{ weights}.$$

The larger layer has

$$600\times400=240{,}000\text{ weights}.$$

Their ratio is

$$\frac{240{,}000}{60{,}000}=4.$$

Equivalently, each dimension gained a factor of $2$, so the product gained $2\times2=2^2=4$. This reasoning counts weights only; optional biases would add one value per output rather than another full pairwise table.

Question 5

Classical dense self-attention increases a sequence from 512 to 1,024 tokens while head width stays fixed. Calculate the query-key score positions before and after, then state what the ratio does and does not prove.

Show the step-by-step solution

For a full dense query-key grid, the position count is $n^2$. Before the increase,

$$512^2=262{,}144.$$

After doubling the length,

$$1{,}024^2=1{,}048{,}576.$$

The ratio is

$$\frac{1{,}048{,}576}{262{,}144}=4.$$

This proves that the full grid has four times as many score positions and that the standard dense attention arithmetic has a quadratic length term when head width is fixed. It does not prove an exact $4\times$ change in measured latency or peak memory, because implementation and hardware behavior are additional variables.

Question 6

A student claims (a + b)² = a² + b². Test the claim with a = 2 and b = 3, then give the correct expansion.

Show the step-by-step solution

Substituting $a=2$ and $b=3$ into the left side gives

$$(2+3)^2=5^2=25.$$

The claimed right side gives

$$2^2+3^2=4+9=13.$$

Because $25\ne13$, one counterexample disproves the claimed identity. The correct expansion follows by distributing both factors:

$$\begin{aligned} (a+b)^2 &=(a+b)(a+b)\\ &=a^2+ab+ba+b^2\\ &=a^2+2ab+b^2. \end{aligned}$$

At $a=2$ and $b=3$, this gives $4+12+9=25$, matching the left side.

Question 7

In the real-number setting, classify 0⁻², √(−9), (−8)¹ᐟ³, √((−5)²), and 0⁰. Give a reason or convention warning for each.

Show the step-by-step solution

Consider each expression with its required domain:

1. $0^{-2}=1/0^2$ would divide by zero, so it is undefined.
2. $\sqrt{-9}$ has no real value because no real number has square $-9$.
3. $(-8)^{1/3}$ can be read as the real cube root $\sqrt[3]{-8}=-2$ because the root index is odd. Floating-point power functions may not preserve this exact rational interpretation.
4. $\sqrt{(-5)^2}=\sqrt{25}=5$. The principal square root is nonnegative, so this equals $|-5|$, not $-5$.
5. $0^0$ is context-dependent. The zero-exponent derivation assumed a nonzero base, so it does not settle this case. A formula or software system must state whether it adopts a value such as $1$ or leaves the expression undefined.

The common lesson is that notation alone does not erase the assumptions used to define the operation.

Sources

1. Mathematics for Machine Learning companion website
2. Mathematics for Machine Learning book PDF
3. NumPy documentation: numpy.power
4. NumPy documentation: numpy.sqrt
5. NumPy documentation: numpy.logspace
6. Attention Is All You Need