Math you need, on one page

This is all the math the course uses. Each idea has one small example with real numbers. Read it once now, or come back when a level sends you here. Each section ends with a short “Try it”: compute the answer on paper first, then open it to check.

e, exp and ln

$e \approx 2.72$ is a fixed number, like $\pi$. $e^x$ (in code np.exp(x)) is “e multiplied by itself x times”. It works for any x, not only whole numbers, and it is always positive.

“Multiplied by itself x times” only makes sense for whole numbers. For x = 0, x = 0.5 or a negative x, just read the table: $e^0 = 1$ and $e^{0.5} \approx 1.65$.

A negative power means “one over”: $e^{-1} = 1/e \approx 0.37$. And $e^0 = 1$.

ln (the natural logarithm, np.log in code) undoes $e^x$: it answers “e to what power gives this number?”. So $\ln 1 = 0$, $\ln e = 1$, $\ln 2 \approx 0.69$ and $\ln 10 \approx 2.30$. ln only takes positive numbers, and ln of a number below 1 is negative.

Four rules cover everything in the course:

Level 2 draws its 3D loss surface with height $\ln(1 + \text{loss})$. Adding 1 keeps the number positive, and ln pulls big values down a lot: $\ln(1 + 0) = 0$, $\ln(1 + 9) \approx 2.30$, $\ln(1 + 99) \approx 4.61$. A bigger loss is still higher.

Used in: level 3 (cross-entropy), level 10 (softmax, perplexity), level 11.

Powers and square roots

$x^2 = x \times x$ and $x^3 = x \times x \times x$. A square is never negative: $(-3)^2 = (-3)\times(-3) = 9$. The square root $\sqrt{x}$ (np.sqrt) undoes a square: $\sqrt{9} = 3$. It always means the positive root.

In Python a power is written **: x ** 2 is $x^2$, and on a NumPy array a ** 2 squares every element. Don't write x ^ 2: in Python ^ means something else, and on floats it gives an error.

Used in: level 2 (squared error), level 6 (the slope of tanh), level 7 (Adam).

Σ, the sum sign

$\sum_{i=0}^{n-1} x_i$ means “add $x_i$ for every i from 0 to n − 1”. The small number under a letter is its position, counted from 0 as in code: $x_0$ is the first x. (Many math books count from 1 instead; the sum is the same.)

Used in: level 1 (each cell of a matrix product is a sum), level 10 (softmax divides by a sum).

Mean, variance and standard deviation

The mean is the sum divided by how many numbers there are. The variance is the mean of the squared distances from the mean. The standard deviation (std) is the square root of the variance: a typical distance from the mean, in the same units as the numbers.

Used in: level 7 (starting weights), level 10 (Gaussian noise), level 16 and N2 (LayerNorm), D1.

The derivative: a slope

The derivative of a function says how fast its output changes when its input changes a tiny bit. It is the slope of its graph at that point. You can measure it with two numbers: change the input a little, divide the change in output by the change in input.

When a function has several inputs, the partial derivative $\partial L / \partial w$ is the slope when only w moves and the other inputs stay fixed. All the partial derivatives together are the gradient.

For a function with one input, the derivative is often written $f'(x)$, read “f prime of x”. For $f(x) = x^2$, $f'(x) = 2x$, so $f'(3) = 6$. A layer’s $f'$ in level 4 is the slope of its activation function.

Used in: level 2 onward. Training is “move each parameter against its slope”.

The chain rule

If y depends on u, and u depends on x, then the slope of y with respect to x is the two slopes multiplied:

Used in: level 2 (where the gradient formula comes from), level 6 (backpropagation is this rule, over and over).

sin, cos and polynomials

sin (np.sin) is a wave: it goes up and down between −1 and 1 forever. $\sin 0 = 0$, and it repeats every $2\pi \approx 6.28$. Level 8 uses $y = \sin(\pi x)$ as a smooth curve to learn: for x from −1 to 1 it goes down to −1, back through 0, up to 1 and back to 0.

cos (np.cos) is the same wave, shifted by a quarter turn: $\cos 0 = 1$. Angles in sin and cos are in radians, not degrees. A full turn is $2\pi \approx 6.28$ radians, and a quarter turn is $\pi/2 \approx 1.57$. So 1 radian is about 57°.

A polynomial is a sum of powers of x, each times a number called a coefficient: $c_0 + c_1 x + c_2 x^2 + \dots$ Its degree is the biggest power. A degree-1 polynomial is a straight line; a higher degree can bend more times.

Used in: level 8 (fitting a curve with too many parameters), level 16 (position encodings use sin and cos), level 19 (RoPE turns vectors by angles in radians).

Very small and very large numbers: 1e−5 and powers of ten

Computers write 1e-5 for $1 \times 10^{-5} = 0.00001$ and 3e4 for $3 \times 10^{4} = 30000$. The number after the “e” says how many places to move the decimal point: a minus sign moves it to the left (a small number), a plus or no sign moves it to the right (a big number). This “e” has nothing to do with $e \approx 2.72$.

To multiply powers of ten, add the exponents: $10^3 \times 10^4 = 10^7$. To divide them, subtract the exponents: $10^{14} / 10^{10} = 10^4$. With a number in front, divide that number separately.

Used in: level 6 (checking a gradient with a tiny step), U2, level 20 (memory and speed of a large model).

Vectors, matrices and shapes

A vector is a list of numbers, like [2, 0]. A matrix is a table of numbers; its shape (rows, columns) says how big it is. Rows and columns count from 0. Matrix multiplication A @ B needs the inner sizes to match: $(n, k)\,@\,(k, m) \to (n, m)$. In formulas, two matrices written next to each other with nothing between them, like $X W$, mean the same product X @ W. Level 1 teaches all of this from scratch.

Used in: level 1 and every level after it.