Level N4 · Foundations · Classic networks · runs in your browser · last part on your computer

LSTM and GRU

How do gates let a network remember for longer?

Side trip · best after level 10 · Probability and sampling

Boss level

Write a complete GRU step in NumPy and pass every test in your browser. Then, on your computer, write the LSTM cell inside boss.py yourself and train it on text we generate until at least 90% of the lines it writes follow every rule.

In level N3, a plain RNN could not remember the first symbol for 40 steps. Every step passed its hidden state through tanh and multiplied it by $w_h$ , so what was added early got smaller and smaller. An LSTM (the name stands for “long short-term memory”) adds a second path, the cell state c, where the default is to keep what is there.

The boss at the end has a part that runs on your computer. If you have not set that up yet, see Run the code on your computer.

1. The cell state: a moving belt

Think of c as an object on a moving belt (a conveyor belt): it travels from step to step, and at each step a little can be removed from it or added to it.

Three gates decide what happens to c at each step. Each gate is a number between 0 and 1:

The forget gate f: how much of the old c to keep.
The input gate i: how much of the new candidate g to add.
The output gate o: how much of c to show as the output h.

c_t = f \cdot c_{t-1} + i \cdot g

h_t = o \cdot \tanh(c_t)

One LSTM cell: c moves along a belt from step to step

Move the gates. The belt’s thickness is the size of the cell state c.

what is left after t steps with nothing new added (i = 0)

Gates (0 to 1) f, keep 0.90 i, add 0.50 o, show 0.60

Inputs c_prev 1.00 g, new candidate 0.80

× f per step: after 10 steps ? is left, after 50 steps ? (a plain RNN, about × 0.5: 0.0010 and 8.9e-16)

c above 0 c below 0 LSTM cell, × f per step plain RNN, about × 0.5 per step

Number cₚᵣₑᵥ = 1.0, f = 0.9, i = 0.5, g = 0.8. What is c = f·cₚᵣₑᵥ + i·g?

🔒 Answer the question above to unlock

Number c = 1.3 and the output gate o = 0.6. Use tanh(1.3) ≈ 0.86. What is h = o·tanh(c)? (2 decimals)

Where do the gates come from? Each one is a small layer with a sigmoid, so its output lands between 0 and 1. It reads the new input and the previous h, just like the RNN did.

Number The forget gate is f = sigmoid(w_f·x + u_f·hₚᵣₑᵥ + b_f). With w_f = 2, x = 1, u_f = 0, hₚᵣₑᵥ = 0 and b_f = 1, what is f? Use e⁻³ ≈ 0.05. (2 decimals)

2. Why the cell keeps its value

🔒 Answer the question above to unlock

Look at the path c takes from one step to the next: multiply by f, add something. There is no tanh and no weight matrix on that path. If nothing new is added, the cell keeps a fraction f of what it had, every step. Move the f slider in the lab and watch the LSTM curve.

Number Nothing new is added (i = 0) and f = 0.9 at every step. c starts at 1. Use 0.9⁵ ≈ 0.59. What is c after 10 steps? (2 decimals)

ChooseNothing new is added. To keep half of c after 50 steps, the forget gate f must be about…

When the loss at a late step sends its signal back to step 1 along the cell, each step multiplies it by f. With f close to 1, the signal arrives. In the plain RNN, each step multiplied it by about 0.5.

One detail decides whether training starts to work at all. Before training, a gate’s output is about sigmoid(its bias), because the weights start small. With the forget bias at 0, f starts at sigmoid(0) = 0.5. With the forget bias at 2, f starts at sigmoid(2) ≈ 0.88. The lab below trained two LSTMs that differ only in this starting bias.

ChooseTwo LSTMs differ only in the starting bias of their forget gate: 0 or 2. The task is “remember the first symbol” with k = 80, 1,500 training steps and a 32-number state. A plain RNN only guesses (about 50%). Which LSTM reaches about 100%?

Remember the first symbol: real training runs

Pick a sequence length k. Try k = 40 and k = 80 to see where the networks start to differ.

Loading runs…

…

RNNLSTM, every bias starts at 0LSTM, forget bias starts at 2 1,500 steps of 64 sequences, a 32-number hidden state, accuracy on 1,000 new sequences

I got stuck here Why does the starting value of the forget bias matter so much?

Before training, a gate’s output is about sigmoid(bias). With bias 0, f starts at 0.5, and over 79 steps the cell path keeps 0.5⁷⁹ ≈ 1.65e-24 of the signal. The network cannot even notice that step 1 matters, so it never learns to raise f.

With bias 2, f starts at 0.8808, and 0.8808⁷⁹ ≈ 4.42e-5. That is small, but it is enough for training to see the connection. Training then pushes f closer to 1 by itself. Starting the forget bias at 1 or 2 is standard practice.

3. The GRU: the same idea with two gates

🔒 Answer the question above to unlock

The LSTM carries two vectors, h and c, and uses three gates. A GRU (gated recurrent unit) keeps the same “keep by default” idea with fewer parts: one state h and two gates.

Both gates are sigmoids, between 0 and 1, each computed from $x_t$ and $h_{t-1}$ with its own weights, like the LSTM’s gates:

z = \sigma(x_t W_z + h_{t-1} U_z + b_z) \qquad r = \sigma(x_t W_r + h_{t-1} U_r + b_r)

The update gate z decides how much of the new candidate $\tilde h$ to take. The rest, 1 − z, is the old h, kept as it was:

h_t = (1 - z)\, h_{t-1} + z\, \tilde h

The reset gate r decides how much of the old h to use when making the candidate. The candidate has its own weights $W_h$ , $U_h$ , $b_h$ :

\tilde h = \tanh\bigl(x_t W_h + (r \cdot h_{t-1})\, U_h + b_h\bigr)

Here $r \cdot h_{t-1}$ multiplies number by number, like r * h in NumPy. With r near 0, the candidate ignores the old h.

Number A GRU step with hₚᵣₑᵥ = 0.6, update gate z = 0.25 and candidate h̃ = −0.2. What is the new h = (1 − z)·hₚᵣₑᵥ + z·h̃?

🔒 Answer the question above to unlock

One GRU step: keep the old state, or take the new candidate

Move the gates. The new h always lands between h_prev and the candidate h̃; z says how close h is to h̃.

Gates (0 to 1) r, reset 1.00 z, update 0.30

Inputs h_prev, old state 0.80 a, input part -0.60

h̃ = tanh(-0.60 + 1.00 × 0.80) = 0.20, and h = 0.70 × 0.80 + 0.30 × 0.20 = 0.62.

old state h_prev candidate h̃ new h the way from h_prev to h̃

The LSTM and the GRU side by side
	LSTM	GRU
state carried between steps	h and c	h only
gates	forget, input, output	update z, reset r
weight blocks	4 (f, i, g, o)	3 (z, r, h̃)
“keep” path	c = f · c + i · g	h = (1 − z) · h + z · h̃

When z is near 0, the old h passes through almost unchanged, like c in the LSTM, so the gradient can also pass back through it. The GRU has no separate output gate and no separate cell, so it has fewer weights.

Number Inputs have D = 10 numbers and the state has H = 20. Each block of a recurrent layer has a (D, H) input matrix, an (H, H) state matrix and H biases: 10×20 + 20×20 + 20 = 620 numbers. An LSTM has 4 blocks. How many numbers does a GRU have?

4. Reading both ways

🔒 Answer the question above to unlock

Every RNN so far read left to right, so the state at a word knows only the words before it. Sometimes the words after matter more. In “he sat by the bank of the river”, it is “river”, four words later, that says which kind of bank this is (a bank for money, or the side of a river).

A bidirectional RNN runs two RNNs over the same sentence: one left to right, one right to left. The output at each position is the two states side by side, so every word sees the whole sentence, from both ends.

Number The sentence is “he sat by the bank of the river” (8 words, counting from 0, “bank” is word 4). The backward RNN starts at “river”. When it reaches “bank”, how many words has it read, counting “bank”?

🔒 Answer the question above to unlock

Reading both ways: what each direction has seen

Tap a word. The forward RNN has read everything up to it; the backward RNN everything after it.

→ forward RNN ← backward RNN output

At “bank”, the forward state has read he sat by the bank; the backward state has read river the of bank. The output for “bank” is both states side by side.

forward RNN, left to right backward RNN, right to left a word that direction has already read

Shape Each direction has H = 32. For an 8-word sentence, the output is one row per word, each row the forward and backward states side by side. What shape is it?

ChooseTwo jobs. (1) Label every word of a finished sentence as noun, verb and so on. (2) Write a sentence one word at a time. Which job can a bidirectional RNN do?

🔒 Answer the question above to unlock

I got stuck here If reading both ways is better, why don’t text generators do it?

A generator writes one word at a time. When it picks word 5, words 6, 7 and 8 do not exist yet, so there is nothing for a right-to-left pass to read. Bidirectional models are for jobs where the whole input is there before you answer: labeling each word, classifying a sentence, or the encoder in level N5, which reads a whole input before the decoder starts writing.

5. Lines that follow the rules

🔒 Answer the question above to unlock

Our own generator writes lines like these:

the gray bee wants the gray cup.
the blue fox sees the blue box.

The rules: the <color> <animal> <verb> the <same color> <thing>. There are 6 colors, 6 animals, 6 verbs and 6 things. A character-level model reads one letter at a time and learns to predict the next one. To follow the last rule, it has to remember the first color for 17 characters, across the animal and the verb.

Number There are 6 colors. If a model gets the grammar right but picks the second color at random, what fraction of its lines follow every rule? (percent, 1 decimal)

Predict firstGuess before you look: the same training (1,800 steps, a 128-number state), but with a plain RNN. What happens to the color rule?

The boss run, played back: what each model writes as it trains

Drag through training. Each line is checked against the rules as you watch.

Loading the recorded samples…

Waiting for the recorded run…

LSTM plain RNN chance level for the color rule the step you picked

The LSTM learns the grammar first: by step 450, 98% of its lines follow the grammar. The colors still differ, at about the 1-in-6 chance level. Then, between step 900 and step 1,200, it suddenly starts carrying the color across. The plain RNN learns the grammar too, but never the color.

I got stuck here Why is there a long flat stretch before the jump?

Getting the grammar right helps at every character, so it is learned first. The color rule helps at only one character per line, and only once the network finds a way to carry the color forward. Until some gate starts holding it, the gradient for that rule is weak. Once a gate starts holding the color, the gain grows quickly, and the fraction of good lines jumps from 17% to 94% in 300 steps.

Gates have limits too. We made the rule harder: the second color comes 40 characters later (for example the big <thing> and the <same color> <thing>). Our LSTM did not learn that version in 6,000 steps. In level 14, attention solves this kind of long-distance rule directly.

6. Write it yourself

Here is one LSTM step with vectors. As everywhere in the course, vectors are rows: the input x has D numbers, h and c have H. All four gates come from one matrix multiply, x @ W + h @ U + b, which gives 4H numbers. The first H belong to the forget gate, the next H to the input gate, then H for the candidate g, then H for the output gate. So W is (D, 4H) and U is (H, 4H). z[:H] takes the first H numbers, z[H:2 * H] the next H, and so on.

Write the line that updates the cell.

CodeWrite the cell update: keep f of the old c, and add i times the candidate g.

Enter keeps the indent · Tab indents · Esc then Tab leaves the editor · ⌘/Ctrl + Enter runs

Go deeper PyTorch orders the gates differently

This page and boss.py order the four blocks f, i, g, o. PyTorch’s built-in nn.LSTM orders them i, f, g, o. If you ever copy weights between the two, swap the first two blocks of W, U and b. The math is the same; only the order of the blocks inside the big matrix differs.

🔒 Answer the question above to unlock

The boss, part 1: a GRU step

Now a cell you have not seen as code. Write one GRU step from the formulas in section 3: the update gate z, the reset gate r, the candidate $\tilde h$ , and the new h. The weights are in a dictionary, so P["Wz"] is $W_z$ .

CodeThe boss, part 1. Write one GRU step: the update gate z, the reset gate r, the candidate h̃ and the new h, all from section 3. Replace ____ with as many lines as you need.

Enter keeps the indent · Tab indents · Esc then Tab leaves the editor · ⌘/Ctrl + Enter runs

🔒 Answer the question above to unlock

The boss, part 2: your LSTM cell on your computer

Download boss.py into a folder of its own and open it there (setup). It has everything for training except one part: LSTMCell.forward. Write it in PyTorch, using the step from section 6 (torch.sigmoid and torch.tanh work like the NumPy ones, and x now holds a whole batch, one row per example, so slice the columns: z[:, :H]).

Then run python boss.py. It first checks your cell against the formulas on this page, then trains a character model that uses it for a few minutes and samples 200 lines. At 180 or more good lines it prints a line that starts with N4 PASS. Paste that line here. The code at the end of the line only shows that you pasted it unchanged; this part relies on your honesty.

The level counts as cleared once part 1 (the GRU step) and this line both pass.

CodeThe boss, part 2. Paste the PASS line that boss.py printed for your own LSTM cell, between the quotes.

Enter keeps the indent · Tab indents · Esc then Tab leaves the editor · ⌘/Ctrl + Enter runs

Then try

Turn your cell into a plain RNN and train again. In forward, right after the line that computes z, add return torch.tanh(z[:, :self.H]), c. Then put a # in front of check_cell() in main, because this cell fails that check on purpose. In our run the LSTM cell wrote 199 good lines out of 200, and the plain RNN wrote 6. Read the three lines it prints and see which rules they break.

Recap

a summary for when you finish the level

The key formulas and common mistakes appear here once you clear the level.

You can now

Compute one LSTM step by hand: the new cell c from f, i and g, then h from o and tanh(c).
Explain why the cell keeps its value: the path from c to c is only “multiply by f, add”.
Write a GRU step in NumPy from its formulas, and an LSTM cell in PyTorch.

Keep in mind

$c_{t} = f \cdot c_{t - 1} + i \cdot g$ and $h_{t} = o \cdot tanh (c_{t})$
With nothing new added (i = 0), c keeps $f^{n}$ of its value after n steps
GRU: $h_{t} = (1 - z) h_{t - 1} + z \tilde{h}$ , with $\tilde{h} = tanh (x_{t} W_{h} + (r \cdot h_{t - 1}) U_{h} + b_{h})$
One LSTM multiply x @ W + h @ U + b gives 4H numbers: blocks f, i, g, o

Common mistakes

Starting the forget bias at 0: f begins at 0.5 and the signal from early steps vanishes before training can use it.
Swapping (1 − z) and z in the GRU update, or multiplying the candidate’s $U_{h}$ by h instead of $r \cdot h$ .

‹ N3. Recurrent networksN5. Seq2seq + attention ›

Press ? for keyboard shortcuts

LSTM and GRU

A short test to skip this level

1. The cell state: a moving belt

One LSTM cell: c moves along a belt from step to step

2. Why the cell keeps its value

Remember the first symbol: real training runs

3. The GRU: the same idea with two gates

One GRU step: keep the old state, or take the new candidate

4. Reading both ways

Reading both ways: what each direction has seen

5. Lines that follow the rules

The boss run, played back: what each model writes as it trains

6. Write it yourself

The boss, part 1: a GRU step

The boss, part 2: your LSTM cell on your computer

Recap

You can now

Keep in mind

Common mistakes

Keyboard shortcuts