Automated Evals and Regression Suites
You can score 25 golden rows with an exact-match assertion, count how often three biased graders agree with that assertion on four pairs, compute the normal interval at z=2 for accuracies 0.80 and 0.64, and turn the logs for fee@2 and fee@3 into an exit code.
- Level
- Navigator
- Time
- 22 h
- Prerequisites
- R01-201

Jump to
After this course you can
The student can build an automated evaluation harness that detects regressions across prompt versions and blocks any candidate prompt whose accuracy drops or latency/cost exceeds fixed thresholds.
Twenty-five rows and an exact match
A golden dataset here is a file of 25 rows, with ids 1 through 25. Each row holds one note and one gold string, the reply that should come back. The harness stores the candidate reply beside that gold string. You grade the stored strings. Once the two strings are fixed, the pass bit is fixed too.
Normalize both strings the same way. Trim spaces at the two ends. Lowercase every letter. Collapse every run of spaces into a single space. The assertion exact_match passes when the two results are the same string, and it fails when they differ. Hits H is the number of rows that pass. Accuracy A is H divided by 25. The gold strings in this file are already lowercase and single-spaced, and you still run the three steps on both sides.
Try two replies against the gold string 40. The reply written with two spaces before 40 and two spaces after it trims to 40, and the later steps leave 40. exact_match passes. The reply The Fee is 40, with two spaces between The and Fee, lowercases and then collapses to the fee is 40. That differs from 40, so exact_match fails. A longer reply can look helpful and still fail. The pass bit uses the strings after this procedure.
The log lists the failing ids, and you recover H by subtraction. fee@1 fails on ids 3, 8, 14, 19, and 22. That is 5 failures. H = 25 minus 5 = 20. A = 20/25 = 0.80. fee@2 fails on ids 3, 5, 8, 11, 14, 16, 19, 22, and 24. That is 9 failures. H = 25 minus 9 = 16. A = 16/25 = 0.64. The four ids that fee@1 passed and fee@2 failed are 5, 11, 16, and 24. Every id that fee@1 failed is still a failure on fee@2. fee@3 fails on the same five ids as fee@1, so H = 20 and A = 0.80. Latency and cost are totals over the same 25 rows. fee@1 latency totals 10000 ms, so the mean is 10000/25 = 400 ms, and the cost total 50.00 gives the mean 2.00. fee@2 latency totals 9000 ms, mean 360 ms, and the cost total 40.00 gives the mean 1.60. fee@3 latency totals 16000 ms, mean 640 ms, and the cost total 62.50 gives the mean 2.50.
Position, length, and the phrase
A model-grader is a second language model that picks a winning reply. Published comparisons of such graders show three leans. Position: the reply placed in slot 1 wins more often. Verbosity: the longer reply wins more often. Self-preference: a reply that resembles the grader's own wording wins more often. The three pencil graders below copy those leans, and self-preference is copied with one fixed phrase, as an assistant. The harness gate uses exact_match. The graders show how a pick can move while the gold string stays put.
Four pairs use a short reply S and a long reply L. Tokens are pieces split on spaces, and punctuation stays on its piece. Pair P1 has gold 40, S equal to 40 (exact, 1 token), and L equal to the fee is 40 (4 tokens, not exact). The key winner is S. Pair P2 has gold lund, S equal to lund (exact, 1 token), and L equal to as an assistant the city is lund (7 tokens, not exact). The key winner is S. Pair P3 has gold the fee is 40, S equal to 40 (1 token, not exact), and L equal to the fee is 40 (4 tokens, exact). The key winner is L. Pair P4 has gold 40, S equal to 41 (1 token, not exact), and L equal to the fee is 41 (4 tokens, not exact). The key is a tie, because both replies fail exact_match.
Order SL puts S in slot 1. Order LS puts L in slot 1. The position grader selects slot 1 on every order and has no tie result. On P1, SL agrees and LS disagrees. On P2, SL agrees and LS disagrees. On P3, SL disagrees and LS agrees. On P4, both orders disagree, because the key is a tie. That is 3 agreements in 8 judgments: P1 with SL, P2 with SL, and P3 with LS. On P2 in order LS, slot 1 is the long reply, the key winner is still S, and all three graders select L, so 0 of 3 agree on that order.
The verbosity grader selects the reply with more tokens, and it selects a tie only when the token counts match. Every long reply above is longer, so verbosity selects L on all four pairs. L is the key winner only on P3. P1, P2, and P4 disagree. Agreements: 1 of 4. The self-preference grader reads the normalized text. If exactly one reply contains the phrase as an assistant, it selects that reply. Otherwise, if exactly one reply passes exact_match, it selects that reply. Otherwise it selects a tie. P2 hits the first rule and selects L, against key S. P1 selects S and agrees. P3 selects L and agrees. P4 selects a tie and agrees. Agreements: 3 of 4, on P1, P3, and P4.
The interval and the three gates
The normal approximation for a proportion uses SE, the square root of A times (1 minus A) divided by n, with n = 25. A textbook interval multiplies SE by 1.96, the normal value that leaves 5 percent of the curve outside the two tails. This lesson multiplies by 2, so each product is easy to check by squaring, and the gate reads those ends. If an end is below 0, replace it with 0. If an end is above 1, replace it with 1. For the two accuracies below, both ends already sit inside 0 and 1, so the replacement changes nothing.
For A = 0.80, A times (1 minus A) = 0.80 times 0.20 = 0.16. Then 0.16/25 = 0.0064, and the square root of 0.0064 is 0.08, because 0.08 times 0.08 = 0.0064. Two times SE = 0.16. The lesson interval runs from 0.80 minus 0.16 = 0.64 to 0.80 plus 0.16 = 0.96. With 1.96 the margin is 1.96 times 0.08 = 0.1568, and the lower end is 0.80 minus 0.1568 = 0.6432. For A = 0.64, 0.64 times 0.36 = 0.2304. Then 0.2304/25 = 0.009216, and the square root is 0.096, because 0.096 times 0.096 = 0.009216. Two times SE = 0.192. The lesson interval runs from 0.640 minus 0.192 = 0.448 to 0.640 plus 0.192 = 0.832. With 1.96 the margin is 1.96 times 0.096 = 0.18816, and the lower end is 0.640 minus 0.18816 = 0.45184.
The gate file prompts/fee/gates.txt has three lines, written before you open a candidate log: acc-lower 0.60, lat-max 500, and cost-max 3.00. G-acc passes when the lesson lower end is at least 0.60. G-lat passes when the mean latency is at most 500 ms. G-cost passes when the mean cost is at most 3.00. fee@1 has lower end 0.64, latency 400, and cost 2.00, so all three pass. fee@2 has lower end 0.448, so G-acc fails, while 360 and 1.60 would pass the other two. fee@3 shares the fee@1 interval, so G-acc passes, cost 2.50 passes, and latency 640 fails G-lat. The products n times A and n times (1 minus A) are 20 and 5 when A is 0.80, and 16 and 9 when A is 0.64. This lesson treats the normal formula as usable when both products are at least 5, and both of these logs qualify.
A continuous-integration regression gate runs that check on every new version. Continuous integration here means the same command runs on each version before anyone updates the pointer. The command is python3 eval_runner.py followed by one /prompter argument. Exit 0 means all three gates passed. Exit 1 means at least one gate failed. fee@2 and fee@3 both exit 1. The pointer file stays on the previous id.
Files under prompts/ and the /prompter line
Versions live under prompts/fee/. The file prompts/fee/1.txt begins with the line id: fee@1, then the line parent: none, then a blank line, then the body. prompts/fee/2.txt has id: fee@2 and parent: fee@1. prompts/fee/3.txt has id: fee@3 and parent: fee@1. prompts/fee/CURRENT holds one line, fee@1. You copy a new id into CURRENT only after that id has exited 0.
The argument /prompter fee@2 splits on @ into the task fee and the version 2. The path is prompts/fee/2.txt. The harness opens that path and requires the first line to be exactly id: fee@2. A file whose first line reads id: fee@9 exits 1 before any row is scored, and H stays uncomputed. The same split builds prompts/fee/3.txt from /prompter fee@3, and prompts/fee/1.txt from /prompter fee@1.
Count body tokens by splitting on spaces. The fee@1 body is four lines: Read the note. (3 tokens), Copy the fee digits. (4), Reply with the digits only. (5), and Stop. (1). The sum is 3 plus 4 plus 5 plus 1 = 13. fee@2 keeps lines 1, 2, and 4, and replaces line 3 with Reply with a short sentence. which is Reply, with, a, short, sentence. and so 5 tokens. The body sum stays 13. fee@3 keeps the four fee@1 lines and adds Check the sum twice. which is Check, the, sum, twice. and so 4 tokens. The body sum is 13 plus 4 = 17. The token sum tells you which lines changed. The exit code comes from H, the latency total, and the cost total. fee@2 replaced the line that asked for digits only, and the log shows H falling from 20 to 16. fee@3 added one line, and the log shows H still 20 with mean latency 640, from the total 16000. The extra tokens are the diff. The mean is the measurement the gate reads.
fee@2 from the file to exit 1
Open prompts/fee/2.txt through the argument /prompter fee@2. The first line is id: fee@2, so scoring runs. The parent line is parent: fee@1. The gate file still reads acc-lower 0.60, lat-max 500, and cost-max 3.00. The failure list is 3, 5, 8, 11, 14, 16, 19, 22, and 24. The count is 9. H = 25 minus 9 = 16. A = 16/25 = 0.64. SE = 0.096, because 0.096 times 0.096 = 0.009216 and 0.2304/25 = 0.009216. The lesson ends are 0.448 and 0.832. The lower end 0.448 is below 0.60, so G-acc fails. The latency total 9000 gives the mean 360, and 360 is at most 500, so G-lat passes. The cost total 40.00 gives the mean 1.60, and 1.60 is at most 3.00, so G-cost passes. One failed gate is enough. The command python3 eval_runner.py /prompter fee@2 exits 1, and CURRENT remains the line fee@1.
Run the same command on fee@3. The path is prompts/fee/3.txt and the id line matches. H stays 20, A stays 0.80, and the lesson interval stays 0.64 to 0.96, so G-acc passes. The cost total 62.50 gives the mean 2.50, so G-cost passes. The latency total 16000 gives the mean 640. Because 640 is above 500, G-lat fails, and python3 eval_runner.py /prompter fee@3 exits 1. The body grew from 13 tokens to 17 because the line Check the sum twice. was added. The block you report is the mean 640 from those 25 rows. Both candidates leave CURRENT on fee@1, since a copy into CURRENT waits for an exit 0.
Grade fee@4 and pair P4
Grade /prompter fee@4. The file prompts/fee/4.txt starts with id: fee@4 and parent: fee@1, then a blank line, then the same four body lines as fee@1. The log fails on ids 3, 8, 14, 19, and 22. The latency total is 8750 ms. The cost total is 80.00. The gate file is unchanged. Write H, A, SE, both lesson ends, the result of each gate, and the exit code. Then, for pair P4, write what the self-preference grader selects and whether that selection agrees with the key. Then write the token count of the fee@2 line Reply with a short sentence.
H = 20, because 25 minus 5 = 20. A = 0.80. SE = 0.08. The lesson ends are 0.64 and 0.96. G-acc passes, because 0.64 is at least 0.60. The mean latency is 8750/25 = 350, so G-lat passes. The mean cost is 80.00/25 = 3.20, and 3.20 is above 3.00, so G-cost fails. The exit code is 1, and CURRENT stays on fee@1. On P4 the self-preference grader selects a tie and agrees with the key. The fee@2 line splits into five pieces, so the token count is 5.
Check questions
fee@1 fails on ids 3, 8, 14, 19, and 22. What are H and A?
20 and 0.80. Five failures leave 20 hits, and 20/25 = 0.80.
How many of the 8 position judgments agree with the key, and how many of the 4 verbosity judgments agree?
3 of 8, and 1 of 4. The position agreements are P1 in order SL, P2 in order SL, and P3 in order LS. Verbosity agrees only on P3.
For A = 0.64 and n = 25, what is SE, what are the two lesson ends, and does G-acc pass?
SE is 0.096. The ends are 0.448 and 0.832. G-acc fails because 0.448 is below 0.60.
Which path does /prompter fee@3 open, what is the mean latency, and what is the exit code?
prompts/fee/3.txt, 640 ms, and exit 1. G-lat fails because 640 is above 500, while G-acc and G-cost pass.
exact_match on the 25 gold strings fixes H, and the lesson interval is A plus or minus 2 times SE. The command exits 1 when the lower end is below 0.60, the mean latency is above 500 ms, or the mean cost is above 3.00. On the four pairs the position grader agrees 3 of 8 times, the verbosity grader agrees 1 of 4, and the self-preference grader agrees 3 of 4.
Final task
Submit an evaluation harness (eval_runner.py) tested against 10 seeded prompt diffs (5 valid improvements, 5 regressions with subtle defects); the runner must classify all 10 diffs accurately in the container test suite.
Terms in this course
This course has no terms of its own in the glossary yet. Terms from earlier courses are used here too.
The first lesson is open to read. The rest of the course is being written, and tests and diplomas come later. Join the waitlist