Skip to content
POLARISAI ACADEMY
R09-301 · Navigator

Post-Training (SFT, RLHF, DPO) and Interpretability

You can mask a four-slot supervised loss so only the answer labels count, turn a reward gap of ln(3) into a preference probability, compute a one-pair DPO loss and its two log-probability gradients, and patch one attention-head output to measure the fraction of a logit gap it restores.

Level
Navigator
Time
28 h (estimate)
Prerequisites
R09-201
Course 3 of 4
Jump to

After this course you can

The student can implement the exact DPO loss function and an activation-patching hook in PyTorch that shifts preference probabilities on a toy model and isolates the target attention head responsible for a seeded circuit behavior.

A mask on the answer labels

Supervised fine-tuning, SFT, keeps the next-token cross-entropy you already compute from a score and a softmax. The new piece is a mask lined up with the labels. The prompt is the text already written in the question. The answer is the text the weight update should follow. In this lesson a mask entry is 1 when the label is an answer token and 0 when the label is a prompt token. The row loss is the sum of each slot loss times its mask entry, divided by the sum of the mask. The forward pass still predicts every next token. The mask decides which of those losses enter the mean. A common library setting does the same job by writing the ignore index -100 on each dropped label. PyTorch cross-entropy skips every -100 and averages the labels that remain.

Take five tokens, Q, 2, +, 4 and E, with ids 10, 11, 12, 13 and 14, and with roles prompt, prompt, prompt, answer, answer. The slot at position i predicts the token at position i+1, so the four labels are 11, 12, 13 and 14, for the tokens 2, +, 4 and E. Their roles are prompt, prompt, answer, answer, and the label mask is 0, 0, 1, 1. Stored labels are then -100, -100, 13, 14. Use the four losses 1.0, 0.6, 0.8 and 0.4, each already equal to the negative log of the correct token. The masked sum is 0.8 + 0.4 = 1.2. The mask sum is 2. The row loss is 1.2 / 2 = 0.6. The unmasked mean of the same four losses is (1.0 + 0.6 + 0.8 + 0.4) / 4 = 2.8 / 4 = 0.7. Dividing the masked sum by all four slots instead gives 1.2 / 4 = 0.3. The lesson rule divides by the count of ones, because that is the mean over the labels that are actually scored.

The mask follows the label, the token being predicted. Position 2 holds +, a prompt token, and the label at that slot is 4, the first answer token, so the mask entry is 1 and the loss 0.8 stays in the sum. A mask copied from the five input roles is 0, 0, 0, 1, 1. The first four of those entries are 0, 0, 0, 1. Multiplied onto 1.0, 0.6, 0.8 and 0.4, that sliced mask keeps only 0.4, and its mean is 0.4. The 0.8 on the first answer token has dropped out. The slip is the one-step gap between the input role at position i and the label role at position i+1. Paste the four losses and the mask 0, 0, 1, 1 into the chat box, and keep the reply only when the mean is 0.6.

Two scores and a preference probability

A reward model reads one prompt and one completion and prints a single score. A preference pair holds two completions of the same prompt: y_w, the completion a person chose, and y_l, the completion they set aside. The Bradley-Terry model turns the score gap into a probability. With the sigmoid σ(z) = 1 / (1 + e to the power -z), the probability that y_w is preferred equals σ(r_w - r_l). The pair loss is the negative log of that probability, which equals ln(1 + e to the power -(r_w - r_l)).

Set r_w = ln(3) and r_l = 0. The gap is ln(3). Then e to the power -ln(3) equals 1/3, and σ(ln(3)) = 1 / (1 + 1/3) = 3/4. The pair loss is -ln(3/4) = ln(4/3), which rounds to 0.288 at three decimals. A second pair with r_w = 1 and r_l = 1 has gap 0, probability 1/2 and loss ln(2), which rounds to 0.693. The equal scores pay a larger loss than the gap ln(3). A training step on this loss raises the chosen score relative to the rejected score.

Reward hacking shows up when the score under training can rise while a separate checker stays at 0. The checker here gives 1 only when the reply is exactly the one token 4, and gives 0 in every other case. The proxy starts at 0, adds 1 when the character 4 appears at least once, and adds 0.2 for every token after the first. Reply A is the single token 4. The checker gives 1. The proxy gives 1 + 0.2 × 0 = 1. Reply B is five tokens, 4 4 4 4 4. The checker gives 0, because the reply is longer than one token. The proxy gives 1 + 0.2 × 4 = 1.8. The proxy ranks B above A, and the checker ranks A above B. The extra 0.8 pays for four repeated tokens. A step that climbs the proxy can collect that 0.8 and leave the exact one-token answer behind.

The DPO loss on one pair

Direct Preference Optimization, DPO, writes a preference loss on the policy's own sequence log-probabilities. A frozen reference, written π_ref, stores the log-probabilities from before this preference update, often the log-probabilities of the SFT weights. The sequence log-probability sums the token log-probabilities on the answer slots only. The prompt slots stay out, for the same reason the SFT mask zeroes them. For one pair, the margin is u = (log π_θ(y_w|x) - log π_ref(y_w|x)) - (log π_θ(y_l|x) - log π_ref(y_l|x)). With a positive coefficient β, set z = β u and take the loss L = -ln σ(z). Under the Bradley-Terry model with a penalty that keeps the policy near the reference, the prompt's shared partition term is the same for both completions, so it cancels in the gap. The gap that remains is z. The pair is then scored from the policy and the frozen reference.

Use β = 1, log π_θ(y_w|x) = -1, log π_ref(y_w|x) = -1, log π_θ(y_l|x) = -1 - ln(3) and log π_ref(y_l|x) = -1. The winner's difference from the reference is 0. The loser's difference from the reference is -ln(3). Then u = 0 - (-ln(3)) = ln(3), and z = ln(3). The probability σ(z) is 3/4, and L = ln(4/3), the same 0.288 as the reward gap ln(3). The reference logs are held fixed. The partial derivative of L with respect to log π_θ(y_w|x) is -β × (1 - σ(z)). Here 1 - 3/4 = 1/4, and σ(-ln(3)) = 1 / (1 + 3) = 1/4, the same weight. The winner's partial is -1 × 1/4 = -0.25. The loser's partial is +0.25. Gradient descent of step 0.1 subtracts 0.1 times the partial. The winner changes by -0.1 × -0.25 = +0.025, from -1 to -0.975. The loser changes by -0.025, from -1 - ln(3) to -1 - ln(3) - 0.025. The new margin is ln(3) + 0.05.

The same u with β = 2 gives z = 2 ln(3) = ln(9). Then e to the power -z equals 1/9, σ(z) = 9/10, and the loss is ln(10/9), which rounds to 0.105. The factor 1 - σ(z) is 1/10, so the two partials are -2 × 0.1 = -0.2 and +0.2. They are smaller in size than -0.25 and +0.25, because the sigmoid moved from 3/4 to 9/10 and the leftover weight shrank by more than the extra factor of β put back. In this preference setting, proximal policy optimization, PPO, trains against a separate reward model and clips the probability ratio between the new policy and the old one. The advantage is the number that says how far a completion's reward sits above a baseline. With clip width ε = 0.2, a ratio of 1.5 and an advantage of 2, the unclipped product is 1.5 × 2 = 3. The clip stops the ratio at 1 + 0.2 = 1.2, and 1.2 × 2 = 2.4. The clipped objective uses the smaller of 3 and 2.4, which is 2.4, because the advantage is positive. The DPO pair above carries the two partials -0.25 and +0.25. That pair keeps a frozen reference inside the margin, and the PPO figures here are a separate clip on a ratio.

Patch one head, read the logit

Activation patching runs two forward passes and then copies one internal write from the clean run into the corrupted run. The clean prompt is 2+2. The corrupted prompt replaces the second 2 with 3, and the logit of the answer token 4 drops. Logit here means the score of token 4 before softmax. This toy uses width 1, so an attention head, one head inside the attention block, writes a single number that is added to a one-number residual. The fixed readout weight is 1/2, and the logit equals that weight times the residual after the head. Every write before the chosen head is identical on the two runs, which is the toy's setup, so the gap sits in this head's output. Residual before the head: 2. Clean head output: 4. Corrupted head output: -1.

The clean residual after the head is 2 + 4 = 6, and the clean logit is 0.5 × 6 = 3. The corrupted residual is 2 + (-1) = 1, and the corrupted logit is 0.5 × 1 = 0.5. The gap is 3 - 0.5 = 2.5. Patch the clean head output into the corrupted run: replace -1 with 4 and leave the residual before the head at 2. The patched residual is 6, and the patched logit is 3. The restored amount is 3 - 0.5 = 2.5. The fraction of the gap is 2.5 / 2.5 = 1. A control head that writes 0 on the clean run and 0 on the corrupted run changes nothing when you copy it. The corrupted logit stays 0.5, the restored amount is 0, and the fraction is 0. Inspecting the heads means computing that fraction one head at a time, with every other head left on its corrupted output. The head with fraction 1 carries the whole gap in this toy. The head with fraction 0 carries a zero share of it.

A linear probe reads the residual and predicts a label. On these two points the residuals after the head are 6 on the clean run and 1 on the corrupted run, with labels yes and no. The midpoint of 6 and 1 is 3.5. The score residual minus 3.5 is 2.5 on the clean point and -2.5 on the corrupted point. A positive score answers yes. Both points land on the matching side of 0, so the probe is right on 2 of 2 points. A rule that always answers yes is right on 1 of 2. The probe places a cut between the two residuals. The patch is the write that moves the corrupted logit from 0.5 to 3.

An example

One prompt, from the mask to the patch

Start from the five tokens with ids 10, 11, 12, 13, 14. The four labels are 11, 12, 13, 14, and the answer mask turns them into -100, -100, 13, 14. The losses 1.0, 0.6, 0.8, 0.4 then give the masked sum 1.2, the mask count 2 and the row loss 0.6. The unmasked mean is 0.7. The sliced input mask 0, 0, 0, 1 keeps only 0.4. On the same prompt, the reward scores ln(3) and 0 give the gap ln(3), the preference probability 3/4 and the pair loss ln(4/3), which rounds to 0.288. Equal scores of 1 and 1 give the loss ln(2), which rounds to 0.693. Reply A, the single token 4, scores 1 on the checker and 1 on the proxy. Reply B, five 4s, scores 0 on the checker and 1.8 on the proxy.

For DPO, β = 1 and the four sequence logs are -1, -1, -1 - ln(3) and -1, in the order policy winner, reference winner, policy loser, reference loser. The margin u equals ln(3), σ(u) equals 3/4, and the loss equals ln(4/3) again. The partials are -0.25 and +0.25. A gradient step of 0.1 moves the winner's log-probability from -1 to -0.975 and the loser's from -1 - ln(3) to -1 - ln(3) - 0.025, and the margin grows from ln(3) to ln(3) + 0.05. With β = 2 the same u gives σ = 9/10, loss ln(10/9) which rounds to 0.105, and partials -0.2 and +0.2. Separately, a PPO ratio of 1.5 with advantage 2 and clip width 0.2 contributes 2.4 after the clip, while the unclipped product is 3.

On the width-1 patch, the residual before the head is 2, the clean head writes 4 and the corrupted head writes -1. The logits are 3 and 0.5. Copying 4 into the corrupted run restores the logit 3, which is the whole gap, fraction 1. Copying the control head's 0 leaves the logit at 0.5, fraction 0. The probe scores are 6 - 3.5 = 2.5 and 1 - 3.5 = -2.5, so both points fall on the matching side of 0. The four written results to check in the chat box are the masked mean 0.6, the preference probability 3/4, the DPO partials -0.25 and +0.25, and the patched logit 3.

Exercise

A second row on paper

Use the same five-token shape, roles prompt, prompt, prompt, answer, answer, and the same four label slots. Take the losses 0.3, 0.9, 1.1 and 0.5. Write the label mask, the masked mean, the unmasked mean, and the mean from the sliced input mask 0, 0, 0, 1. Set the reward scores to ln(4) and 0. Write the preference probability and the pair loss rounded to three decimals. With the same checker and the same proxy, score reply C, the single token 4, and reply D, the three tokens 4, 9, 9. For DPO use β = 1, policy and reference log-probability -2 on the winner, reference log-probability -2 on the loser, and policy log-probability -2 - ln(4) on the loser. Write the loss rounded to three decimals and the two partial derivatives. For the patch set the residual before the head to 4, the clean head output to 6, the corrupted head output to 0 and the readout to 1/2. Write the clean logit, the corrupted logit, the patched logit and the fraction. From the two residuals after the head, write the probe scores using the midpoint of those residuals.

The label mask is 0, 0, 1, 1. The masked sum is 1.6 and the masked mean is 0.8. The unmasked mean is 2.8 / 4 = 0.7. The sliced mask keeps 0.5, so that mean is 0.5. The gap ln(4) gives probability 4/5 = 0.8 and loss ln(5/4), which rounds to 0.223. Reply C scores 1 on the checker and 1 on the proxy. Reply D scores 0 on the checker and 1 + 0.2 × 2 = 1.4 on the proxy. The DPO margin is ln(4), the loss rounds to 0.223, and the partials are -0.2 and +0.2. The clean logit is 5, the corrupted logit is 2, the patched logit is 5, and the fraction is 1. The residuals after the head are 10 and 4, the midpoint is 7, and the probe scores are 3 and -3.

Check questions

Answer on your own first. Then open a question to see the answer.

For the losses 1.0, 0.6, 0.8 and 0.4, what is the masked mean under the label mask 0, 0, 1, 1, and what mean does the sliced input mask 0, 0, 0, 1 give?

0.6 under the label mask, and 0.4 under the sliced input mask. The sliced mask drops the 0.8 on the first answer token.

With reward scores ln(3) and 0, what is the Bradley-Terry probability, and what is the pair loss rounded to three decimals?

The probability is 3/4. The pair loss is ln(4/3), which rounds to 0.288.

On the worked DPO pair with β = 1 and margin ln(3), what are the two partial derivatives of the loss with respect to the policy log-probabilities of the winner and the loser?

-0.25 for the winner and +0.25 for the loser. The shared weight is 1 - 3/4 = 1/4, and β is 1.

The corrupted logit starts at 0.5. What is it after the clean head output 4 replaces the corrupted head output -1, and what fraction of the gap does that restore?

The logit becomes 3, and the fraction is 1. The control head that writes 0 on both runs restores a fraction of 0.

Take with you

The masked mean scores the answer labels, a Bradley-Terry gap of ln(3) gives probability 3/4 and loss ln(4/3), DPO turns that gap into the partials -0.25 and +0.25 against a frozen reference, and copying the clean head output moves the toy logit from 0.5 to 3.

Final task

Submit dpo_loss.py and activation_patch.py that pass 25 fixed unit tests verifying exact DPO loss/gradient agreement against reference equations and deterministic identification of the seeded induction head in a frozen tiny transformer.

The final task is marked against fixed answer keys and automated tests once the tests open.

Terms in this course

This course has no terms of its own in the glossary yet. Terms from earlier courses are used here too.

More lessons coming

The first lesson is open to read. The rest of the course is being written, and tests and diplomas come later. Join the waitlist