Skip to content
POLARISAI ACADEMY
R08-401 · Navigator

Benchmark Integrity and Air-Gapped Local Deployment

You can mark a public item when at least two of its four-word windows sit in the train text, ship the prompt with the higher private exact rate, drop a judge pair that flips with the order, and count cache blocks, draft tokens, and dropped packets on a closed host.

Level
Navigator
Time
26 h (estimate)
Prerequisites
R08-301, R03-201
Course 4 of 4
Jump to

After this course you can

The student can detect benchmark contamination in a supplied evaluation set and package a complete air-gapped local inference and retrieval service that passes all functional tests under strict firewall drop rules.

Two windows and the item has leaked

A public leaderboard can rise because the test items already sat in the training text. That leak is called contamination. This lesson normalizes a line by lowercasing it, deleting every character that is not a letter or a space, collapsing repeated spaces, and trimming the ends. You then cut the line into windows of four words, stepping one word at a time. An item is contaminated when at least two of those windows occur as windows in the train text, or when the whole normalized item occurs as a continuous stretch of that text. A full audit uses longer windows and the whole training collection. These two rules are short enough to redo on paper, and one hit on either rule is enough to keep the item out of a rate you quote.

The train text in this lesson is the single line many books name the capital of france in a list and the lazy dog sleeps after lunch. Public item P1 is name the capital of france in one word. Its five windows are name the capital of, the capital of france, capital of france in, of france in one, and france in one word. The first three occur in the train text. The train text has of france in a, so of france in one misses, and france in one word misses. Three hits meet the rule of two, so P1 is contaminated. The full P1 line is absent as a stretch, because the train text continues with a list. P2 is sort the three bills by due day. Its four windows are sort the three bills, the three bills by, three bills by due, and bills by due day. The word sort never occurs in the train text, so all four miss. P3 is a red fox jumps the lazy dog. Its four windows are a red fox jumps, red fox jumps the, fox jumps the lazy, and jumps the lazy dog. The train window the lazy dog sleeps shares three words with the last of those and differs on the fourth, so it misses. P2 and P3 are clean. One item of the three is contaminated, which is 1/3.

The public five and the private five

Once a public exact-match rate is the number a team chases, later changes get picked to move that number. That is Goodhart's law in the form used on scores: when the measure becomes the target, it stops measuring the job. The public board is five items. The private set is five other items, written after training and never published. Exact match on an item is 1 when the reply string equals the key, and 0 otherwise. The rate is the sum divided by 5. This lesson treats a public rate that leads the private rate by 0.10 or more as a number you do not steer by. You report the private rate beside the public one, and you keep the prompt with the higher private rate.

Prompt A scores 4 exact hits on the public five and 2 on the private five. The rates are 4/5 = 0.80 and 2/5 = 0.40. The gap is 0.80 − 0.40 = 0.40, which clears 0.10. Prompt B scores 3 and 3, so both rates are 3/5 = 0.60, and the gap is 0. The public comparison picks A, because 0.80 is above 0.60. The private comparison picks B, because 0.60 is above 0.40. You ship B. The domain items are invoice orders, so the key is a sort you can redo. Due day decides, and a tie goes to the lower id. The worked item has id 1 on day 5, id 2 on day 2, and id 3 on day 2. Gold order is 2, 3, 1. The three pairs that have to hold are 2 before 3, 2 before 1, and 3 before 1. A reply of 2, 1, 3 keeps the first two pairs and places 1 before 3, so the third pair fails. The pair score is 2/3 and the exact score is 0. A reply of 2, 3, 1 keeps all three pairs, so the pair score is 3/3 and the exact score is 1. The board still reports the exact rate. The pair score shows that the reply was one swap from the key. The same-day sheet is five invoices you sort by hand, scored both ways, with a second five kept off the public board.

Swap the order, then count the facts

A judge model can prefer the answer shown first. This lesson shows every pair twice. H is the answer a human prefers, and W is the other answer. If both orders name the same winner, the pair stays and that winner is the judge's call. If the winner flips, the pair drops and adds nothing to the tally. On item 1 both orders pick H. On item 2 the judge picks H when H is first and W when W is first. On item 3 both orders pick W. On item 4 both orders pick H. Items 1, 3, and 4 stay. Item 2 drops. Three of the four stay. Among those three, the judge agrees with the human on items 1 and 4 and disagrees on item 3, so agreement on the kept pairs is 2/3. A tally that used only the first-shown order would have counted item 2 as a win for H. The swap takes that count out.

Length pulls the same judge along a second path. Four replies each contain the same three required facts, so each fact score is 3 of 3. Their lengths are 30, 60, 120, and 240 tokens. The judge awards 2, 3, 4, and 5. Each time the length doubles, the judge adds 1 point, while the fact score stays 3. This lesson keeps the fact score and sets the judge points aside when the fact scores tie. The reply you show is the shortest tied reply, which has 30 tokens.

Blocks, a draft, and a closed host

PagedAttention, paging for the KV cache, cuts that cache into blocks of a fixed token count. This lesson uses 16 tokens in a block. A request holds the next whole number of blocks at or above peak length divided by 16, and the empty slots in its last block are the waste inside that request. R1 peaks at 40 tokens. 40/16 = 2.5, so R1 holds 3 blocks, 48 slots, and 48 − 40 = 8 empty slots. R2 peaks at 17. 17/16 = 1.0625, so R2 holds 2 blocks, 32 slots, and 32 − 17 = 15 empty slots. R3 peaks at 16. 16/16 = 1, so R3 holds 1 block and 0 empty slots. Together they hold 3 + 2 + 1 = 6 blocks, which is 96 slots. Reserving a 128-token context for each request would hold 3 × 128 = 384 slots. The paged layout holds 384 − 96 = 288 fewer slots. Continuous batching fills the running set at each iteration: a request that has emitted its last new token returns its blocks, and a waiting request that fits can enter before the next iteration. The pool has 8 blocks. R1 must emit 3 new tokens and R2 must emit 1. They start together on 5 blocks, leaving 3 free. After iteration 1, R2 is done and frees its 2 blocks. R1 still holds 3, so 5 are free. R3 needs 1 block and must emit 2 new tokens, so it enters before iteration 2. Iteration 2 runs R1 and R3. Iteration 3 finishes both. The queue is empty after 3 iterations. A static schedule that finishes R1 and R2 before starting R3 spends max(3, 1) = 3 iterations on the first batch and 2 on R3, which is 5. The continuous run is shorter by 5 − 3 = 2 iterations.

Speculative decoding lets a small draft model propose tokens that a larger target model checks. This lesson checks at temperature 0 with 3 proposed tokens. Accept from the left while the draft token equals the target's greedy token. The round always emits one target token where it stops: the target token at the first mismatch, or one extra target token when all 3 match. Round 1, draft red fox jumps against target red fox runs, emits red fox runs, 3 tokens. Round 2, draft over the dog matches the target, and the extra token is sleeps, 4 tokens. Round 3, draft in a barn against target under a tree, emits under, 1 token. Round 4, draft at noon sharp against target at dusk sharp, emits at dusk, 2 tokens. The four rounds emit 3 + 4 + 1 + 2 = 10 tokens across 4 target passes, which is 10/4 = 2.5 tokens per target pass. Lesson prices, chosen so the ratio stays on paper: one draft proposal of 3 tokens costs 1 unit, and one target pass costs 4. Four rounds cost 4 × (1 + 4) = 20. Ten plain target passes would cost 10 × 4 = 40. The lesson speedup is 40/20 = 2. A production draft pays a small forward pass for each proposed token, so the price 1 stands in for those three small passes together.

On a closed host, an air-gapped machine, this lesson's IPv4 log allows a packet only when the destination is 127.0.0.1. Every other address is dropped, including a public address, an address on the local network, and a DNS resolver. The weights and the vector index are already files on disk. The log has 10 attempts. Six stay on 127.0.0.1: health on port 8080, a vector list on port 6333, two generations at temperature 0 on port 8080, a vector query on port 6333, and a grounded generation on port 8080. Four leave the host: 1.1.1.1 port 53, 93.184.216.34 port 443, 192.168.1.50 port 11434, and 10.0.0.8 port 443. The ports are the ones written in this log. The closed-host rule passes when the drop counter equals 4 and five checks pass. F1, the health body is the string ok. F2, the two temperature-0 replies are both the string due day 2. F3, the vector query returns one row, id 17, with the text invoice 17 due day 2. F4, the grounded reply contains both 17 and day 2. F5, the drop counter is 4. If the query returns no row because an embedding call tried 93.184.216.34 and was dropped, the counter can read 4 while F3 fails. Point the embedding step at the model file on disk and run the same log again. Those four destinations stay dropped, and F3 then returns id 17.

An example

One board, one queue, one closed log

Start from the train line many books name the capital of france in a list and the lazy dog sleeps after lunch. P1 contributes the windows name the capital of, the capital of france, and capital of france in, three hits, and the windows of france in one and france in one word miss. P2 contributes four windows and zero hits, because sort is absent from the train line. P3 contributes four windows and zero hits, because jumps the lazy dog differs from the lazy dog sleeps. The contaminated count is 1 of 3, which is 1/3. Prompt A is 4/5 = 0.80 on the public five and 2/5 = 0.40 on the private five, gap 0.40. Prompt B is 3/5 = 0.60 on both. Ship B.

The invoice key is 2, 3, 1. The reply 2, 1, 3 keeps 2 before 3 and 2 before 1, and it loses 3 before 1, so the pair score is 2/3 and the exact score is 0. The reply 2, 3, 1 scores 3/3 and exact 1. Judge items 1, 3, and 4 stay, item 2 drops, and 2/3 of the kept pairs agree with the human. The four fact-tied replies score 3, 3, 3, and 3, and the one you show has 30 tokens. R1, R2, and R3 hold 3, 2, and 1 blocks, together 6, with 8, 15, and 0 empty slots. The continuous queue is empty after 3 iterations. The static schedule takes 5. The four draft rounds emit 3, 4, 1, and 2 tokens, together 10, at a lesson cost of 20 against 40 for ten plain target passes. The log allows the six calls to 127.0.0.1 and drops the other four. With embeddings read from disk, F1 through F5 pass, and the drop counter reads 4.

Exercise

Check it on paper

Item P4 is the capital of france in june. Write its three windows, how many of them occur in the train line, and whether P4 is contaminated. Prompt C scores 5 exact hits on the public five and 2 on the private five. Write both rates, the gap, and which of B and C you ship. Score the reply 3, 2, 1 against gold 2, 3, 1 as a pair score and an exact score. Item 5 picks H when H is first and W when W is first. Write whether it stays. R4 peaks at 33 tokens. Write its block count and its empty slots. A fifth draft round matches all 3 proposed tokens. Write how many tokens that round emits. One extra attempt goes to 172.16.0.4 port 443. Write the new drop count.

P4's windows are the capital of france, capital of france in, and of france in june. The first two occur in the train line and the third misses, so the hit count is 2 and P4 is contaminated. The full P4 line is absent as a stretch, because the train line has in a, where P4 has in june. Prompt C is 5/5 = 1.00 and 2/5 = 0.40, and the gap is 0.60. Ship B, whose private rate 0.60 is above C's 0.40. The reply 3, 2, 1 loses 2 before 3 and keeps 2 before 1 and 3 before 1, so the pair score is 2/3 and the exact score is 0. Item 5 drops. R4 holds 3 blocks, because 33/16 = 2.0625, and the empty slots are 48 − 33 = 15. The fifth round emits 4 tokens. The new drop count is 4 + 1 = 5.

Check questions

Answer on your own first. Then open a question to see the answer.

How many four-word windows of P1 occur in the train line, and what fraction of the three public items is contaminated?

Three windows occur. The fraction is 1/3.

Which prompt do you ship, what is its private exact rate, and what pair score does the reply 2, 1, 3 get against gold 2, 3, 1?

Ship B. The private rate is 0.60. The pair score is 2/3.

Which of the four judge items drops, how many stay, and how many tokens does the reply you show contain when all four fact scores are 3?

Item 2 drops. Three stay. The shown reply has 30 tokens.

How many blocks do R1, R2, and R3 hold together, after how many iterations is the continuous queue empty, how many tokens do the four draft rounds emit, and how many of the ten log attempts are dropped?

They hold 6 blocks. The queue is empty after 3 iterations. The rounds emit 10 tokens. 4 attempts are dropped.

Take with you

Quote a public exact rate beside a private set you can recompute, after leaked items are marked and judge pairs that flip with order are dropped. On the closed host, cache blocks, draft tokens, and dropped packets are three counts you can rerun from the same written rules.

Final task

Complete a keyed contamination audit of a seeded benchmark dataset and submit a containerised local AI + retrieval stack that passes 30 end-to-end functional tests with outbound networking completely disabled.

The final task is marked against fixed answer keys and automated tests once the tests open.

Terms in this course

This course has no terms of its own in the glossary yet. Terms from earlier courses are used here too.

More lessons coming

The first lesson is open to read. The rest of the course is being written, and tests and diplomas come later. Join the waitlist