Building a Transformer from Scratch in PyTorch
You can count three BPE merges on a three-line corpus, stack the embedding rows for a cut line, compute both causal attention outputs for a width-1 key, and check that a shared quarter-turn leaves a dot product unchanged.
- Level
- Navigator
- Time
- 32 h
- Prerequisites
- R09-101

Jump to
After this course you can
The student can implement a complete BPE tokeniser and decoder-only Transformer with RoPE and KV-caching from scratch in PyTorch (without nn.Transformer) that matches reference forward/backward tensors and KV-cache equivalence.
Counting a BPE merge
Byte-pair encoding, BPE, builds the token list by repeatedly merging the adjacent pair that occurs most often. The merge rule is the same one a byte-level tokenizer uses. This hand version starts from the letters written below, so every count stays visible. A byte-level run would start from bytes in place of these letters. The corpus is three lines, already split into symbols, and the end mark · is a symbol in the same way a letter is. Line A is a a b ·. Line B is a a b ·. Line C is a b ·. A pair is two neighboring symbols. Line A contributes the pairs aa, ab and b· once each. Line B contributes those same three pairs once each. Line C contributes ab and b·. The totals are aa = 2, ab = 3 and b· = 3.
The merge is the pair with the largest total. ab and b· tie at 3. The tie rule compares the written pair alphabetically, the first symbol and then the second. The symbol a comes before b, so the first merge is ab, stored as one new symbol ab. In a pass, scan left to right and replace every non-overlapping match, then continue after the symbol you just wrote. These three lines have no overlap, so each a that sits immediately before a b simply joins it. After the merge the lines are A: a ab ·, B: a ab · and C: ab ·. The new totals are a followed by ab = 2, and ab followed by · = 3. The second merge is ab·. The lines are now A: a ab·, B: a ab· and C: ab·. One pair remains, a followed by ab·, with total 2, and the third merge is aab·. Lines A and B each become the single symbol aab·. Line C stays ab·, because no a sits in front of it. The saved merge list, in order, is ab, then ab·, then aab·.
Encoding applies that saved list and does not count the new line. Start from characters and run the three merges in the saved order. The training line a a b · becomes a ab ·, then a ab·, then aab·, which is one token. The line a b · becomes ab ·, then ab·, and the third merge finds nothing to join, so the line is the single token ab·. The line a a a b · is outside the corpus. The first merge joins the a that touches b and leaves a a ab ·. The second merge produces a a ab·. The third merge joins the a that touches ab·, and the leftmost a has no partner, leaving the two tokens a and aab·. You can paste the three training lines into the chat box, ask for the pair totals, and keep the answer only when it matches aa = 2, ab = 3 and b· = 3.
A row for each token id
Each symbol the merges can emit gets an integer id and a row of numbers called an embedding. The ids follow one stated order: the starting symbols a, b and ·, then the merges in the order they were learned. id 0 is a with the row [1, 0]. id 1 is b with the row [4, 1]. id 2 is · with the row [1, 3]. id 3 is ab with the row [2, 2]. id 4 is ab· with the row [0, 1]. id 5 is aab· with the row [3, 1]. A line that became one token is one row. The symbol aab· is id 5, so the array is the single row [3, 1]. A line that became two tokens stacks two rows in that order. The cut a then aab· uses ids 0 and 5, so the array is [[1, 0], [3, 1]]. The row is a lookup. The lookup carries no position. Position enters later, and only on the query and the key.
The table also feeds the dot product you already use as a score. Row a dotted with row b is 1 times 4 plus 0 times 1, which is 4. Row ab· dotted with row aab· is 0 times 3 plus 1 times 1, which is 1. Two rows of different widths have no dot product, so every row in this table has width 2. In the array code the table is a matrix with one row per id, and the line is a list of ids used as indexes. For the attention numbers in the next section the two tokens are a and ab·, ids 0 and 4. The stacked array X is [[1, 0], [0, 1]]. Those two rows are copied from the table. The other four rows stay in the table and stay out of this particular score.
Scaled scores and a causal mask
Scaled dot-product attention builds one score between every query and every key, divides by the square root of the key width, and turns each row of scores into weights with softmax. Here the key width d_k is 1, so the square root is 1 and the scale leaves every product unchanged. The query matrix W_Q is 2 by 1 and equals [[1], [2]]. The key matrix W_K is 2 by 1 and equals [[0], [1]]. The value matrix W_V is 2 by 1 and equals [[2], [8]]. X is 2 by 2. The inner sizes match, and each product is 2 by 1. The query column Q has the entries 1 times 1 plus 0 times 2 = 1, and 0 times 1 plus 1 times 2 = 2. The key column K has the entries 1 times 0 plus 0 times 1 = 0, and 0 times 0 plus 1 times 1 = 1. The value column V has the entries 2 and 8. The score in row i and column j is q_i times k_j. The score matrix is [[0, 1], [0, 2]].
Use the stated exponentials e to the 0 equals 1, e to the 1 equals 2.718 and e to the 2 equals 7.389, and round each probability to three decimals. Softmax exponentiates a row and divides by the sum of that row. Row 0 has masses 1 and 2.718. Their sum is 3.718. Then 1 divided by 3.718 rounds to 0.269, and 2.718 divided by 3.718 rounds to 0.731. The two rounded weights add to 1.000. Row 1 has masses 1 and 7.389. Their sum is 8.389. Then 1 divided by 8.389 rounds to 0.119, and 7.389 divided by 8.389 rounds to 0.881. Those rounded weights also add to 1.000. The output of a token is the weighted sum of the values. Token 0 gives 0.269 times 2 plus 0.731 times 8 = 0.538 plus 5.848 = 6.386. Token 1 gives 0.119 times 2 plus 0.881 times 8 = 0.238 plus 7.048 = 7.286.
A causal mask keeps a token from reading a later value. On paper, drop each future score and put weight 0 on it. In the array code, write a large negative in that future slot before softmax. The value used here is -10 to the 9. Its exponential is 0 in ordinary float32, so the weight on that slot is 0, matching the paper rule. On row 0 the future score is removed, the weight on token 0 is 1, and the weight on token 1 is 0. The causal output is 1 times 2 plus 0 times 8 = 2. Row 1 has no future token, so its causal output stays 7.286. Change the later value from 8 to 80. The causal output of token 0 stays 2. The unmasked output of token 0 becomes 0.269 times 2 plus 0.731 times 80 = 0.538 plus 58.480 = 59.018. The mask is the line that keeps 80 out of the first output. The five lines you write before any training loop are: count the pairs, save the merges, stack the rows, multiply by W_Q, W_K and W_V, then softmax each row after the future slots hold -10 to the 9.
A quarter-turn for position
RoPE, rotary positional embedding, folds the position into the query and the key by rotating pairs of entries. The values stay unrotated. The width-1 keys above have no pair to rotate, so this section uses a separate width-2 pair and a lesson rule with a clean quarter-turn. One quarter-turn counterclockwise sends [x, y] to [-y, x]. Position 0 applies zero turns, and the pair stays [x, y]. Position 1 applies one turn. Position 2 applies two turns and sends [x, y] to [-x, -y]. Position 3 applies three turns and sends [x, y] to [y, -x]. Take the content query [3, 1] and the content key [1, 2]. Both at position 0 give the score 3 times 1 plus 1 times 2 = 5. Move only the key to position 1. The key becomes [-2, 1], and the score against the unturned query is 3 times -2 plus 1 times 1 = -5.
The same offset shows up one step later. Put the query at position 1 and the key at position 2. The query becomes [-1, 3]. The key becomes [-1, -2]. The score is -1 times -1 plus 3 times -2 = 1 minus 6 = -5. The pair of positions (0, 1) and the pair (1, 2) share the offset 1 and share the score -5. Now put both at position 1. The query is [-1, 3], the key is [-2, 1], and the score is -1 times -2 plus 3 times 1 = 5. Both at position 0 and both at position 1 give the score 5. A shared quarter-turn on a single pair leaves the dot product unchanged, and a shared offset repeats the score. The library formula uses a smaller angle. Pair number i at position m is turned by m times 10000 to the power -2i/d, in radians. For the single pair in width 2, i is 0, and 10000 to the power 0 is 1, so the step is 1 radian per position. At position 0 the angle is 0. With cos 0 = 1 and sin 0 = 0 the pair stays put, which matches the lesson rule at position 0. At position 1 the stated four-decimal values are cos 1 = 0.5403 and sin 1 = 0.8415. They turn [1, 0] into [0.5403, 0.8415]. The squares are 0.29192409 and 0.70812225. Their sum is 1.00004634, which still rounds to 1.0000 at four decimals. The turn changes the direction and keeps the length.
RMSNorm and the residual add
RMSNorm divides a row by the root-mean-square of that same row, then multiplies entry by entry by a learned scale. For width 2 the mean uses the divisor 2. Take the row [0, 4]. The squares are 0 and 16, the sum is 16, and the mean is 8. The root of 8 is 2 times the square root of 2. With the stated square root of 2 equal to 1.414, that root is 2.828. Dividing [0, 4] by 2.828 gives [0, 1.414], because 0 divided by 2.828 is 0 and 4 divided by (2 times the square root of 2) equals the square root of 2. The displayed quotient 4 divided by 2.828 equals 1.414 when it is rounded to three decimals. A scale γ = [1, 2] then gives [0, 2.828]. The row [1, 1] is the exact check on the same rule. Its squares sum to 2, the mean is 1, and the root is 1, so the division leaves [1, 1]. With γ = [2, 3] that row becomes [2, 3].
A residual add writes the input row back onto the updated row, entry by entry. Start from u = [0, 4], run RMSNorm with γ = [1, 1], and get [0, 1.414]. The sum is [0, 5.414]. The 4 from the input is still inside the sum. On the attention side, the same add needs matching widths, so the width-1 causal outputs 2 and 7.286 are a separate column. They are added only to a value column of height 2, for example [2, 7.286] plus [1, 1] equals [3, 8.286]. The add uses no product and no mask. It sits after the masked mix, and the earlier token still carries only its own value into that mix.
From three lines to two causal outputs
Count the corpus again in one pass. Line A a a b · gives aa, ab, b·. Line B gives the same three. Line C a b · gives ab, b·. Totals: aa = 2, ab = 3, b· = 3. The alphabetical tie picks ab. The lines become a ab ·, a ab · and ab ·. The next totals are 2 for a+ab and 3 for ab+·, so the second merge is ab·. The lines become a ab·, a ab· and ab·. The last total is 2 for a+ab·, so the third merge is aab·. Encode a a b · by that list and you hold one token, aab·, id 5, row [3, 1]. Encode a a a b · and you hold two tokens, a and aab·, ids 0 and 5, rows [[1, 0], [3, 1]].
For the score, take the tokens a and ab·. Their rows from the table are [1, 0] and [0, 1], so X = [[1, 0], [0, 1]]. With W_Q = [[1], [2]], W_K = [[0], [1]] and W_V = [[2], [8]], the products are Q = [[1], [2]], K = [[0], [1]] and V = [[2], [8]]. The scores are [[0, 1], [0, 2]]. Row 0 uses masses 1 and 2.718, sum 3.718, weights 0.269 and 0.731, unmasked output 0.269 times 2 plus 0.731 times 8 = 6.386. Row 1 uses masses 1 and 7.389, sum 8.389, weights 0.119 and 0.881, output 0.119 times 2 plus 0.881 times 8 = 7.286.
The causal rule puts -10 to the 9 in the future slot of row 0, which sets that weight to 0 on paper and in float32. Token 0 then outputs 2. Token 1 still outputs 7.286. Replacing 8 with 80 leaves the causal output of token 0 at 2, while the unmasked output of token 0 moves to 59.018. Separately, the content query [3, 1] and the content key [1, 2] score 5 when both sit at position 0. With the key alone at position 1 the score is -5, and with the query at position 1 and the key at position 2 the score is -5 again. Both at position 1 score 5.
Close the block on a width-2 row that never entered the attention product. RMSNorm on [0, 4] divides by 2.828 and yields [0, 1.414]. The scale [1, 1] leaves that row, and the residual add gives [0, 5.414]. The same five written lines, merges, rows, three products, a masked softmax and the residual add, are enough to fill one forward step by hand.
Three cuts and one score on paper
Using the same three training lines, the same tie rule and the same three merges, write the tokens, the ids and the stacked rows for the fresh line b a ·. Then write the tokens for b a b ·. Using the worked X, W_Q, W_K, W_V and the stated exponentials, write the causal output of token 0 and of token 1, and write the unmasked output of token 0 after the later value changes from 8 to 80. Last, take the content query [2, 0] and the content key [0, 5]. Write the score with both at position 0, the score with only the key at position 1, and the score with both at position 1. Also divide the row [1, 1] by its root-mean-square and then apply γ = [2, 3].
The line b a · contains none of the merged pairs, so the tokens are b, a and ·, the ids are 1, 0 and 2, and the rows are [4, 1], [1, 0] and [1, 3]. The line b a b · holds the pair a b. The first merge writes b ab ·, the second writes b ab·, and the third finds no a before ab·. The tokens are b and ab·, ids 1 and 4, and the rows are [4, 1] and [0, 1]. The causal outputs are 2 and 7.286. After the later value becomes 80, the unmasked output of token 0 is 59.018 and the causal output of token 0 stays 2. The three scores are 0, -10 and 0. The row [1, 1] has root-mean-square 1, and γ = [2, 3] sends it to [2, 3].
Check questions
On the first pair count, what are the totals for aa, ab and b·, and which pair is merged first?
The totals are 2, 3 and 3. The tie rule merges ab first.
How many tokens does the line a a a b · become under the three saved merges, and what are those tokens?
Two tokens: a and aab·.
With V equal to 2 then 8, what is the causal output of token 0, and what is that causal output after the later value changes from 8 to 80?
It is 2, and it stays 2. The unmasked output moves to 59.018, while the mask holds the first output at 2.
For the content query [3, 1] and the content key [1, 2], what is the score with both at position 0, and what is the score with the query at position 1 and the key at position 2?
5 with both at position 0, and -5 with the query at position 1 and the key at position 2.
A saved merge list cuts a line into ids, a table turns each id into a row, causal scaled attention mixes a token only with its own value and earlier values, and one shared quarter-turn on a pair leaves the dot product unchanged.
Final task
Submit bpe.py and transformer.py tested by 35 fixed unit tests verifying: exact BPE merge reproduction, causal mask zero-leakage of future tokens, exact tensor match against reference weights (< 1e-5), and identical logits with and without KV-cache enabled.
Terms in this course
This course has no terms of its own in the glossary yet. Terms from earlier courses are used here too.
The first lesson is open to read. The rest of the course is being written, and tests and diplomas come later. Join the waitlist