Logarithms for AI: Keep Tiny Probability Products from Vanishing

A small language model assigns probability $0.1$ to each chosen token in a simplified sequence-scoring example. A sequence with 400 such choices has the mathematically nonzero product

Add one more equally likely token and the product is $10^{-401}$. Those are different positive numbers, yet NumPy stores both products as 0.0 when it calculates them with 64-bit floating-point numbers. A ranking based on the stored products has lost the difference.

Logarithms preserve a more useful representation: the two products become the finite sums $400\ln(0.1)\approx-921.034037$ and $401\ln(0.1)\approx-923.336622$. By the end of this lesson, you will be able to read a logarithm as an inverse power, derive the laws that turn products into sums, calculate a short sequence score by hand, and implement the same idea without pretending that log space removes every numerical limit.

This lesson reverses the exponent operation developed in Lesson 7: Powers for AI, whose canonical English route belongs to the course’s four-locale Translation Set. The official Mathematics for Machine Learning companion divides its path into mathematical foundations and the machine-learning problems that use them. Its introductory chapter motivates building foundations before relying on probabilistic models; the logarithm mathematics below is an original probability bridge rather than a definition attributed to Chapter 1.

The missing answer is an exponent

A logarithm asks one precise question: which exponent produces this positive number from the chosen base?

For example, $\log_2(8)=3$ because $2^3=8$, while $\log_{10}(0.01)=-2$ because $10^{-2}=0.01$. The negative answer is an exponent, not a negative input.

Three bases appear often in computing:

Some fields write log without a base, but that convention is not universal. This lesson writes $\ln$ for the natural logarithm and includes a subscript for other bases. NumPy follows the same practical choice: numpy.log computes the natural logarithm element by element.

The inverse relationship gives two checks whenever both expressions are in their real domains:

Multiplication leaves an additive trail

The product law is not a pattern to memorize in isolation. It falls directly out of the exponent law from Lesson 7.

Take positive numbers $a$ and $c$, and use a valid logarithm base $b>0$, $b\ne1$. Let

The definition says $a=b^u$ and $c=b^v$. Multiplying and applying the same-base exponent law gives

The exponent that produces $ac$ is therefore $u+v$:

Division subtracts exponents under the same assumptions:

Raising a positive number to a real power $r$ multiplies its exponent:

Positivity is doing real work in all three laws: $a>0$ and $c>0$ keep every real logarithm defined, and $c>0$ also keeps the quotient denominator nonzero.

Addition does not follow the product law. For a quick counterexample,

but

So $\log(a+c)\ne\log(a)+\log(c)$ in general. A stable operation for adding values that are already stored as logarithms appears later in this lesson, and it is deliberately different from product-to-sum conversion.

Four token choices, worked in both representations

Suppose two candidate continuations each contain four token choices. For this small calculation, treat each listed $p_i$ as the model’s positive conditional probability for the chosen token at position $i$. Multiplying the four factors produces a sequence score; adding their natural logarithms produces the corresponding log-probability. For a fixed observed sequence under the model, these scores are also called its likelihood and log-likelihood. Later lessons will develop the full probability and estimation machinery; here the job is only to calculate and compare the two representations.

Candidate A has twice Candidate B’s product. Its log score is also larger: $-7.824046>-8.517193$. That ordering is guaranteed because $\ln$ has base $e>1$ and is strictly increasing. “Larger” among negative log scores means closer to zero. The score did not become a probability after the logarithm; it became a comparison-friendly representation of one.

Binary64 reaches zero while the log score keeps moving

Computers cannot store every real number. JavaScript’s Number and NumPy’s float64 use the binary64 floating-point format. NumPy reports its machine limits through numpy.finfo: the smallest positive normal float64 value is about $2.2250738585072014\times10^{-308}$. Positive subnormal values fill part of the gap between that number and zero, but with reduced precision; the smallest positive subnormal is about $4.9406564584124654\times10^{-324}$. numpy.nextafter confirms that this is the next representable float after zero in the positive direction.

Move either slider with a pointer or arrow keys. The upper panel follows direct binary64 multiplication through the normal region, the narrow subnormal strip, and zero. The lower panel follows the finite additive value $n\ln(p)$. The solid line with a circle and the dashed line with a square remain distinguishable without color, and the table exposes the selected state and transition rows.

At the server-rendered starting state, $p=0.10$ and $n=400$. Direct multiplication has already produced zero, but $400\ln(0.10)=-921.034037\ldots$ is finite. The log value retains an additive score for the positive factors. Calling $\exp(-921.034037\ldots)$ in float64 still returns zero, because changing representation cannot create a positive float64 value below the format’s range.

The precise transition depends on the data type, values, and order of floating- point operations. The explorer deliberately models repeated binary64 multiplication; it is not a universal cutoff for every probability program.

NumPy mirrors the safe path

The following program was run with NumPy 2.5.2. Direct multiplication uses np.prod, while log-space accumulation uses the element-wise natural logarithm documented by numpy.log followed by a sum.

import numpy as np

for n in (323, 324, 400, 401):
    factors = np.full(n, 0.1, dtype=np.float64)
    direct = np.prod(factors)
    log_score = np.log(factors).sum()
    print(f"n={n}: direct={direct:.4e}, log_score={log_score:.6f}")

limits = np.finfo(np.float64)
print(f"smallest normal:    {limits.smallest_normal:.4e}")
print(f"smallest subnormal: {limits.smallest_subnormal:.4e}")
print(f"next after zero:    {np.nextafter(0.0, 1.0):.4e}")

n=323: direct=9.8813e-324, log_score=-743.734985
n=324: direct=0.0000e+00, log_score=-746.037570
n=400: direct=0.0000e+00, log_score=-921.034037
n=401: direct=0.0000e+00, log_score=-923.336622
smallest normal:    2.2251e-308
smallest subnormal: 4.9407e-324
next after zero:    4.9407e-324

The 400- and 401-factor products collide at stored zero, while their log scores remain ordered. In practice, a system that needs to compare candidate products can keep the log scores and choose the larger one without exponentiating each score. This is the narrow numerical benefit: positive multiplicative factors become finite additive scores. It does not repair invalid inputs, rounding in every other operation, or a final conversion whose result is too small for the chosen type.

Adding alternative probabilities requires another identity. If $x=\ln(a)$ and $y=\ln(c)$, then

not $x+y$. NumPy’s numpy.logaddexp computes that log-of-a-sum directly. For two already logged alternatives, $x=-1000$ and $y=-1001$:

x, y = -1000.0, -1001.0

with np.errstate(divide="ignore"):
    naive = np.log(np.exp(x) + np.exp(y))

stable = np.logaddexp(x, y)
print(naive)
print(stable)
print(np.exp(stable))

-inf
-999.6867383124818
0.0

The naive route exponentiates too early, turning both alternatives into zero. np.logaddexp preserves their finite combined log score. Exponentiating that score still underflows, exactly as the final line shows. Use sum(log_factors) for a product of positive factors; use logaddexp when adding alternatives that are already represented by log values.

Boundaries worth making explicit

The real logarithm’s domain and base conditions prevent several common bugs:

For real arrays, NumPy follows floating-point conventions around this boundary: np.log(0.0) returns -inf and signals a divide-by-zero condition, while np.log(-1.0) returns nan and signals an invalid operation. These are useful machine values, not permission to erase the mathematical distinction. A factor that is genuinely probability zero contributes $-\infty$ to the log score; a positive product that rounded to floating-point zero should have been logged before multiplication. Once a computed product is already zero, np.log(product) cannot reconstruct the lost factors.

Silently replacing zeros with a small positive constant also changes the model’s stated values. If clipping is an intentional modeling decision, record the rule rather than presenting it as exact arithmetic.

Finally, ranking direction depends on the base. For every base $b>1$—including $e$, $2$, and $10$—the logarithm is increasing, so the larger positive product has the larger log value. A valid base between zero and one gives a decreasing logarithm and reverses that ordering. Log-probability implementations normally use $e>1$, which is why the less-negative log score ranks higher.

A finite score can be the useful result

The opening sequences did not need their microscopic probabilities converted back to decimal form to remain distinguishable. Their log scores preserved the multiplicative comparison as ordinary addition, and the product, quotient, and power laws explain why that representation works.

This foundation will support later lessons on probability, likelihood-based estimation, entropy, and stable model calculations. The next curriculum step changes focus to equations as constraints and the sets of values that satisfy them. For now, the practical stopping point is exact: keep positive factors in log space while combining and comparing them, and return to probability space only when the result is both necessary and representable.

Check your understanding

combined = np.logaddexp(-1000.0, -1001.0)
# -999.6867383124818