Skip to content
POLARISAI ACADEMY
R09-401 · Navigator

Reading and Reproducing a Machine Learning Paper

You can recompute a headline gain against the strongest baseline in the same table, apply a drop rule of 0.05 to a two-term ablation, run one seeded step of a written update, and mark a rerun inside or outside a declared tolerance of 0.02.

Level
Navigator
Time
35 h (estimate)
Prerequisites
R09-301
Course 4 of 4
Jump to

After this course you can

The student can implement the core algorithm and ablation experiment from a supplied research paper specification and reproduce its target empirical metrics within a declared tolerance envelope.

The headline and the strongest baseline

A paper packet in this lesson is a short stack of printed tables plus one equation, small enough to audit on paper the same day. The headline quotes a gain over the baseline. Cherry-picking, in this audit, means the quoted baseline is a weaker row in the paper's own table while a stronger row sits on the same page. Read every baseline accuracy in that table and subtract it from the proposed method. The fair gain is the smallest of those differences, the gain against the strongest listed baseline. The overstatement is the headline gain minus that fair gain. You write both numbers, and you name which row the headline used.

The packet's test table, counts out of 100, prints Ours 80, weak baseline W 62, and standard baseline S 74. The headline says the method sits 18 points above the baseline, and 80 - 62 = 18, so the headline used W. The strongest listed baseline is S at 74. The fair gain is 80 - 74 = 6 points, which is accuracy 0.06. The overstatement is 18 - 6 = 12 points. A chat reply that repeats 0.18 has copied the headline. The audit line you keep is 0.06, with W named as the weaker comparison.

A removed term, and a choice read off the test set

An ablation removes one term and measures the drop. Drop equals the full score minus the score without that term. This packet's rule, stated before you look at the runs, gives a term the mark matters only when the drop is at least 0.05. The appendix prints three validation counts out of 100: full model 80, without term C 71, without term D 78. Drop C is 80 - 71 = 9, which is 0.09 and clears 0.05. Drop D is 80 - 78 = 2, which is 0.02 and falls short of 0.05. The sentence both added terms matter clears the rule for C and fails it for D. A paper that prints only the full model against nothing has left this table out, so that sentence has no per-term drop behind it.

Test-set tuning means a setting was chosen by reading the test score. The packet lists four learning rates with validation and test counts out of 100. Rate 0.1 has validation 70 and test 66. Rate 0.01 has validation 76 and test 80. Rate 0.001 has validation 78 and test 77. Rate 0.0001 has validation 73 and test 71. The largest test count is 80, at rate 0.01. The largest validation count is 78, at rate 0.001, and that row's test count is 77. Choosing 0.01 because the test count is 80 uses the test set as a search set. The validation rule freezes 0.001 before the test column is read, and the test accuracy to report is 77/100 = 0.77. The protocol gap is 0.80 - 0.77 = 0.03.

Four lines for one update

The packet writes the update as theta left-arrow theta + eta times (y - theta). Translating it means four lines that use those symbols in that order, with no extra term. Line 1 stores eta = 0.5. Line 2 stores theta = 0.20. Line 3 stores the residual r = y - theta with y = 0.80, so r = 0.80 - 0.20 = 0.60. Line 4 replaces theta with theta + eta times r, which is 0.20 + 0.5 times 0.60 = 0.20 + 0.30 = 0.50. A second step with the same y uses residual 0.80 - 0.50 = 0.30 and lands on 0.50 + 0.5 times 0.30 = 0.65. The same arithmetic is (1 - eta) times theta + eta times y, and 0.5 times 0.20 + 0.5 times 0.80 = 0.50, which matches line 4.

A sign that flips the update to theta left-arrow theta - eta times (y - theta) is a different equation. With the same three numbers the one step is 0.20 - 0.30 = -0.10. The audit compares the code's operator with the printed arrow before any accuracy is discussed. When you paste the equation and the four lines into the chat box, keep the reply only when it prints theta = 0.50 after one step and theta = 0.65 after two. A reply that prints -0.10 has flipped the sign.

A seed and a fixed prefix

A seed is the start value of the packet's generator. This packet names the recurrence so a hand check is exact: x starts at the seed, and each next value is (5x + 3) mod 16. The same seed yields the same list. Set the seed before the first draw, and write it in the report. The paper lists the prefix of four states, including the seed: 1, 8, 11, 10. Check from the left. Start at 1. (5 times 1 + 3) mod 16 = 8. (5 times 8 + 3) = 43, and 43 mod 16 = 11, because 2 times 16 = 32 and 43 - 32 = 11. (5 times 11 + 3) = 58, and 58 mod 16 = 10, because 3 times 16 = 48 and 58 - 48 = 10. The prefix matches seed 1.

Seed 2 produces a different prefix. (5 times 2 + 3) mod 16 = 13. (5 times 13 + 3) = 68, and 68 mod 16 = 4, because 4 times 16 = 64 and 68 - 64 = 4. (5 times 4 + 3) = 23, and 23 mod 16 = 7. The four states are 2, 13, 4, 7. A log that begins with that prefix ran a different stream. Record the seed mismatch before you compare accuracies. A match on 1, 8, 11, 10 says the stream followed the packet. It still leaves the test count to be checked on its own line.

The gap and the envelope

A deviation report has four fields: the printed accuracy, the rerun accuracy, the absolute gap, and the envelope. The absolute gap is the positive difference of the two accuracies. The envelope in this packet is 0.02, so the verdict is inside when the gap is at most 0.02, and outside when the gap is larger. The packet prints 80 correct out of 100, accuracy 0.80. The rerun follows the validation rule, seed 1, and the four update lines, and scores 77 correct out of 100, accuracy 0.77. The gap is the absolute value of 0.77 - 0.80, which is 0.03. Because 0.03 is larger than 0.02, the verdict is outside.

That 0.03 equals the protocol gap from the learning-rate table, 0.80 at the test-chosen rate minus 0.77 at the validation-chosen rate. The report names that cause beside the verdict. A second metric can land inside while this one lands outside. A rerun of drop C that prints 0.08 against the packet's 0.09 has gap 0.01, and 0.01 is inside the same envelope of 0.02. Each metric gets its own line. You leave 0.77 written as 0.77.

An example

One packet, from the headline to the verdict

Read the test table first: Ours 80/100, W 62/100, S 74/100. The headline gain is 80 - 62 = 18 points, the fair gain is 80 - 74 = 6 points, and the overstatement is 12 points. Then the ablation counts on validation: full 80, without C 71, without D 78. The drops are 9 and 2 points. With the rule of 5 points, C receives the mark and D does not. Then the four learning rates. Validation counts 70, 76, 78, 73 select rate 0.001, and that row's test count is 77, so the honest test accuracy is 0.77. The test counts 66, 80, 77, 71 select rate 0.01 and the printed 80. The protocol gap is 80 - 77 = 3 counts, which is 0.03 in accuracy.

The update theta left-arrow theta + eta times (y - theta), with eta = 0.5, theta = 0.20 and y = 0.80, gives residual 0.60 and theta = 0.50 after one step, then residual 0.30 and theta = 0.65 after two. The flipped sign gives -0.10 after one step. The generator (5x + 3) mod 16 from seed 1 lists 1, 8, 11, 10. From seed 2 it lists 2, 13, 4, 7. The deviation line uses printed 80/100 and rerun 77/100. The absolute gap is 0.03, the envelope is 0.02, and the verdict is outside, with the cause named as the test-chosen learning rate. A drop-C rerun of 0.08 against 0.09 has gap 0.01 and the verdict inside.

The audit order on paper is fixed: fair gain, per-term drops, validation choice, four code lines, seed prefix, then one verdict line per metric. Paste that order into the chat box with the packet's tables. Keep the reply when the fair gain is 0.06, the honest test accuracy is 0.77, the one-step theta is 0.50, the prefix is 1, 8, 11, 10, and the accuracy verdict is outside.

Exercise

A second packet on paper

The second packet prints Ours 90/100, W 70/100, S 85/100, and a headline gain of 20 points. Write the fair gain and the overstatement. Ablation counts on validation: full 90, without C 86, without D 81. Write both drops and say which term clears a mark threshold of 5 points. Learning rates, counts out of 100: rate 0.1 has validation 60 and test 64, rate 0.01 has validation 81 and test 79, rate 0.001 has validation 75 and test 83. Write the validation-chosen rate, its test accuracy, the test-chosen rate, and the protocol gap. Run theta left-arrow theta + eta times (y - theta) with eta = 0.25, theta = 0.40 and y = 0.80, and write theta after one step. From seed 4 of the same generator, (5x + 3) mod 16, write the four-state prefix. The printed accuracy is 79/100 and the rerun is 78/100, with the same envelope 0.02. Write the absolute gap and the verdict.

The fair gain is 90 - 85 = 5 points, and the overstatement is 20 - 5 = 15 points. Drop C is 90 - 86 = 4, which falls short of 5, so C misses the mark. Drop D is 90 - 81 = 9, which clears it. Validation counts 60, 81, 75 select rate 0.01, and that row's test accuracy is 79/100 = 0.79. The test counts 64, 79, 83 select rate 0.001, and the protocol gap is 0.83 - 0.79 = 0.04. The residual is 0.80 - 0.40 = 0.40, and theta becomes 0.40 + 0.25 times 0.40 = 0.50. Seed 4 gives (5 times 4 + 3) mod 16 = 7, then (5 times 7 + 3) = 38 and 38 mod 16 = 6, then (5 times 6 + 3) = 33 and 33 mod 16 = 1, so the prefix is 4, 7, 6, 1. The absolute gap is the absolute value of 0.78 - 0.79, which is 0.01, and the verdict is inside the envelope.

Check questions

Answer on your own first. Then open a question to see the answer.

In the first packet, the headline uses W. What is the headline gain, and what is the fair gain against S?

The headline gain is 0.18, from 80/100 minus 62/100. The fair gain against S is 0.06, from 80/100 minus 74/100.

Which learning rate does the validation rule select in the first packet, and what test accuracy belongs to that row?

Rate 0.001, with test accuracy 0.77 from 77/100. The validation counts are 70, 76, 78 and 73, and 78 is the largest.

With eta = 0.5, theta = 0.20 and y = 0.80, what is theta after one step of the printed update, and what is theta after that same step if the plus is written as a minus?

0.50 with the printed plus, because the residual is 0.60 and 0.20 + 0.30 = 0.50. The minus gives 0.20 - 0.30 = -0.10.

Seed 1 of x left-arrow (5x + 3) mod 16 has which four-state prefix, and is the rerun 77/100 against printed 80/100 inside the envelope 0.02?

The prefix is 1, 8, 11, 10. The absolute gap is 0.03, so the verdict is outside the envelope.

Take with you

The audit writes the fair gain against the strongest baseline in the table, a per-term drop against the rule 0.05, the test accuracy on the validation-chosen row, four lines matched to the printed update, the seed prefix, and the absolute gap beside the envelope. On this packet those figures are 0.06, drops 0.09 and 0.02, test accuracy 0.77, theta = 0.50, prefix 1, 8, 11, 10, and a gap of 0.03 marked outside 0.02.

Final task

Complete a 20-item keyed methodological audit of a supplied paper packet and submit a reproduction script (reproduce.py) that runs on a frozen dataset in the test container and reproduces the keyed baseline and ablation metrics within declared tolerances. (Optional Portfolio item: full replication report).

The final task is marked against fixed answer keys and automated tests once the tests open.

Terms in this course

This course has no terms of its own in the glossary yet. Terms from earlier courses are used here too.

More lessons coming

The first lesson is open to read. The rest of the course is being written, and tests and diplomas come later. Join the waitlist