Dataset Curation, Cleaning and Decontamination
You can choose a prompt, retrieval, or a fine-tune from three token counts, then run six fixed steps that drop the wrong license, the exact copy, the near copy, the broken dialogue, and any train row that shares a four-word span with the sealed test.
- Level
- Navigator
- Time
- 24 h

Jump to
After this course you can
The student can build a deterministic dataset curation pipeline that deduplicates rows, strips PII, validates ChatML formatting, and enforces zero n-gram contamination between train and test splits.
Prompt, retrieval, or a fine-tune
A chat reply uses the tokens in the window and weights that stay fixed for that reply. When the missing piece is text, this lesson puts it in one of three places. The window in every count below holds 8,000 tokens. Job P is a price list that changes each week: 40 lines at 20 tokens, plus a 40-token question, which is 800 + 40 = 840 tokens. The list fits, so it goes in the prompt and you replace the text when the week changes. A fine-tune would store one week inside the weights and would keep quoting that week after the list had changed.
Job R is a handbook of 200 pages at 400 tokens a page, which is 80,000 tokens. Dividing by the window gives 80,000 / 8,000 = 10, so the book fills ten windows and cannot be pasted into one prompt. Retrieval-augmented generation, called RAG, leaves the book on disk and copies the four passages that match the question. Four passages at 400 tokens plus the 40-token question use 1,600 + 40 = 1,640 tokens. The rest of the handbook, 80,000 - 1,600 = 78,400 tokens, stays on disk. Job F is a label that will stay fixed for a year, with 2,000 checked dialogues at 200 tokens each. One pass reads 2,000 times 200 = 400,000 tokens, and 400,000 / 8,000 = 50, so the file fills 50 windows. That file is a fine-tune: further training on your rows. The same-day stand-in, while you still have only a handful of rows, is 12 dialogues in the prompt at 80 tokens each, which is 12 times 80 = 960 tokens. This lesson's rule is the one you can apply today: changing text that fits in the window goes in the prompt, text that overflows the window is retrieved as passages, and a stable behavior with at least 2,000 checked rows is a fine-tune.
Licenses and the exact fingerprint
Each row carries one license label, and the label decides whether the row may enter a file you release under CC BY and sell. CC0 may enter, and CC0 asks for no credit line. CC BY may enter, and each CC BY row that passes this step adds one credit line naming the source. CC BY-NC stays out of every file you release under CC BY, sold or unsold, because CC BY lets the next user use the work commercially and the NC term forbids that. To keep CC BY-NC rows, release that file under a non-commercial license such as CC BY-NC. CC BY-SA stays out of a release that is CC BY only: share-alike means a derivative that includes those rows is released under CC BY-SA or another license the share-alike clause names. Leaving the rows out is how this file stays CC BY. A pile of 100 rows holds 40 CC0, 30 CC BY, 20 CC BY-NC, and 10 CC BY-SA. The commercial file keeps 40 + 30 = 70 rows, drops 20 + 10 = 30 rows, and writes 30 credit lines. The unsold notebook, while it is CC BY, also drops the 20 CC BY-NC rows, so the kept count stays 40 + 30 = 70, and the 10 CC BY-SA rows stay out. A notebook that keeps those 20 CC BY-NC rows is released under CC BY-NC, and that release holds 70 + 20 = 90 rows, with the 10 CC BY-SA rows still out.
Exact copies are removed with a checksum you can redo by hand. A real pipeline hashes the normalized line with a cryptographic hash. Here the hand checksum is enough, and a matching checksum still requires a matching string. Lowercase the user line. Delete every character that is not a letter, a digit, or a space. Collapse repeated spaces and trim the ends. The letter-sum uses a=1 through z=26, a space as 0, and a digit as its face value. The fingerprint is (letter-sum mod 97) times 1,000, plus the length. The cat sat on the mat sums as the 33, cat 24, sat 40, on 29, the 33, mat 34. Then 33 + 24 + 40 = 97, 97 + 29 = 126, 126 + 33 = 159, and 159 + 34 = 193. The length is 22. Then 193 - 97 = 96, so the fingerprint is 96022. The same normalized string later in the file drops. The cat sat on a mat replaces the second the (33) with a (1), so the sum is 193 - 33 + 1 = 161 and the length is 20. Then 161 - 97 = 64, and the fingerprint is 64020. The cat sat on the mat today adds today, which is 20 + 15 + 4 + 1 + 25 = 65, so the sum is 193 + 65 = 258 and the length is 28. Then 258 - 2 times 97 = 258 - 194 = 64, and the fingerprint is 64028. The two lines share a remainder and differ in length, so both survive the exact step. Equal fingerprints with different strings are a collision: keep both.
Near copies and the two PII markers
A near copy changes a few words, and the exact step keeps it when the normalized strings differ. This lesson cuts the normalized user line into shingles of three words and looks each shingle up in one table: 1 the cat sat, 2 cat sat on, 3 sat on the, 4 on the mat, 5 sat on a, 6 on a mat, 7 the mat today. Four hash functions use modulus 11: h1(x) = (x + 1) mod 11, h2(x) = (3x + 2) mod 11, h3(x) = (5x + 4) mod 11, and h4(x) = (7x + 3) mod 11. The signature is the four column minimums. The estimate is the fraction of slots that match an earlier kept row. Drop the later row when the estimate is at least 1/2, which means at least 2 of 4 slots. These four hashes are for hand arithmetic. A production MinHash hashes every shingle, with no lookup table, and it uses many more than four functions so the fraction sits closer to the true overlap.
Row A, the cat sat on the mat, uses codes 1, 2, 3, 4. The hash rows are (2, 5, 9, 10), (3, 8, 3, 6), (4, 0, 8, 2), and (5, 3, 2, 9). The minimums are 2, 0, 2, 2. Row B, the cat sat on the mat today, uses 1, 2, 3, 4, 7. Code 7 hashes to (8, 1, 6, 8). Compare each number with the current minimum: 8 against 2, 1 against 0, 6 against 2, and 8 against 2. None is smaller, so B keeps the signature 2, 0, 2, 2. All 4 slots match, the estimate is 4/4, and B drops. The true Jaccard ratio is the shared codes over the union: 4 shared and 5 in the union, so 4/5. The estimate 4/4 sits above 4/5, and both are at least 1/2, so the drop agrees with the ratio. Row C, the cat sat on a mat, uses 1, 2, 5, 6. The hash rows are (2, 5, 9, 10), (3, 8, 3, 6), (6, 6, 7, 5), and (7, 9, 1, 1). The minimums are 2, 5, 1, 1. Against A's 2, 0, 2, 2 only the first slot matches, so the estimate is 1/4 and C stays. The true ratio is 2 shared over a union of 6, which is 2/6. A line whose shingles are all absent from the table skips this drop. Personal data is scrubbed on the rows that remain, in place. An email is one whitespace token that contains @ and a dot after the @, and that token becomes [EMAIL]. A phone is a run of digits, spaces, and hyphens whose digits join to length 10. The run 555-010-1234 splits into 555, 010, and 1234, which join to 5550101234, length 10, and the run becomes [PHONE]. The number 42 has length 2 and stays. Email ada@workshop.example about order 42 becomes Email [EMAIL] about order 42 and adds one [EMAIL]. Call 555-010-1234 before noon becomes Call [PHONE] before noon and adds one [PHONE].
ChatML, synthetic rows, and the sealed test
A dialogue in this lesson is three turns in the order system, user, assistant, stored as ChatML. Each turn begins with the marker <|im_start|>, then the role word, then the text, and it ends with <|im_end|>. Three checks drop a row. The start count differs from the end count. The roles appear in any order other than system, then user, then assistant. The assistant text is empty, as in a turn whose text between the role and <|im_end|> has no words. The token count treats each marker as one token and every other whitespace-separated word as one token, with commas and periods left on the word. The worked dialogue is <|im_start|>system, then You label a ticket as billing, access, or other., then <|im_end|>, then <|im_start|>user, then I was charged twice., then <|im_end|>, then <|im_start|>assistant, then billing, then <|im_end|>. The six markers plus the 17 words system, You, label, a, ticket, as, billing, access, or, other., user, I, was, charged, twice., assistant, billing give 6 + 17 = 23 tokens. The allowed assistant labels in the synthetic file are billing, access, and other.
A synthetic row is a dialogue drafted by a model and then scored with four checks, each 0 or 1: the ChatML checks pass, the assistant label is in that set of three, the user line has at least four words, and the assistant text matches the hand key for that line. Two hard drops run first: the normalized user line already sits in the human file, or any four-word span of the line also sits in the sealed test. Among the rows that remain, keep a sum of at least 3. On the six rows the bits and the hard facts are these. S1 is 1 1 1 1, new, no shared span, sum 4, keep. S2 is 1 1 1 0, new, no shared span, sum 3, keep. S3 is 1 1 0 0, new, no shared span, sum 2, drop. S4 is 1 1 1 1, the same normalized string as a human row, sum 4, drop on the hard rule. S5 is 1 1 1 1, new, no shared span, sum 4, keep. S6 is 0 1 0 0, new, no shared span, sum 1, drop. Hard drops remove S4. Of the five left, S3 and S6 fail the sum. Three rows stay: S1, S2, and S5. The sealed test is written before the split and then left unchanged. A train row drops when any four-word span of a test line appears in that train row's normalized raw user line, the line saved before the scrub, so a [PHONE] marker cannot hide a span. The test line the cat sat on the mat has three spans: the cat sat on, cat sat on the, and sat on the mat. A train line with those same words contains all three, so that train row drops and the test line stays. After that drop the shared count is 0.
Eight rows through the six steps
The file below is released under CC BY for a product you sell. Rows are read in id order, and the lower id is the one that stays. R1 is CC0, user line The cat sat on the mat., fingerprint 96022, shingles 1 2 3 4, valid ChatML. R2 is CC BY, user line the cat sat on the mat, the same normalized string as R1. R3 is CC BY, user line The cat sat on the mat today., fingerprint 64028, shingles 1 2 3 4 7. R4 is CC BY-NC, user line Pay the invoice today. R5 is CC0, user line Call 555-010-1234 before noon. R6 is CC BY-SA, user line Share alike notes from the workshop. R7 is CC BY, user line Email ada@workshop.example about order 42., and its dialogue follows the specimen above with that user line in place of I was charged twice., so 18 words plus 6 markers give 24 tokens, with assistant text billing. R8 is CC0, user line Reset the router and wait., with 3 start markers and 2 end markers. The sealed test has two lines and is never edited: T1 is the cat sat on the mat, and T2 is a quiet closet holds spare cable.
Step 1, license: R4 and R6 drop. Six rows remain: R1, R2, R3, R5, R7, R8. The credit lines are the CC BY rows that passed, R2, R3, and R7, so 3 lines. Step 2, exact string: R2 matches R1's normalized line and drops. Five remain. R3's fingerprint 64028 differs from R1's 96022, so R3 stays. Step 3, near copy: R3 is row B against R1 as row A, the estimate is 4/4, and R3 drops. R5, R7, and R8 have no shingle in the table, so they stay. Four remain. Step 4, scrub: R5's stored line becomes Call [PHONE] before noon, and R7's stored line becomes Email [EMAIL] about order 42. The file now holds one [PHONE] and one [EMAIL]. The raw lines stay on the side for the leak step. No row drops. Step 5, ChatML: R8 has 3 starts and 2 ends, so R8 drops. R1, R5, and R7 each have 3 starts, 3 ends, the role order system then user then assistant, and a non-empty assistant. Three remain. Step 6, leak: T1's three spans all sit in R1's raw line, so R1 drops. T2's spans are a quiet closet holds, quiet closet holds spare, and closet holds spare cable. None of those sits in R5 or R7. The shipped train file holds 2 rows, R5 and R7, and it shares 0 four-word spans with the sealed test. The counts left after the six steps are 6, 5, 4, 4, 3, and 2.
Check it on paper
A returns policy is 6 pages at 400 tokens a page and changes every month. The question is 40 tokens. Say which job it follows, how many tokens go in the window, and how many tokens of the 8,000 remain free. On the pile of 100 rows, write how many the commercial CC BY release keeps and how many credit lines it writes. Recompute the fingerprint of Call 555-010-1234 before noon from the letter groups call 28, digits 26, before 51, and noon 58. Write the signature of row A, the estimate and the true Jaccard for A against B, and the estimate and the true Jaccard for A against C. On the eight-row file, write how many rows remain after each of the six steps, which two ids ship, how many credit lines step 1 writes, and how many four-word spans the shipped file shares with the test. Write how many of the six synthetic rows stay.
The policy follows Job P. The page total is 6 times 400 = 2,400, plus 40 is 2,440 tokens in the window, and 8,000 - 2,440 = 5,560 tokens stay free. The commercial pile keeps 70 rows and writes 30 credit lines. The phone sum is 28 + 26 + 51 + 58 = 163. The length is call 4, plus a space, plus 10 digits, plus a space, plus before 6, plus a space, plus noon 4, which is 27. Then 163 - 97 = 66, and the fingerprint is 66027. Row A's signature is 2, 0, 2, 2. A against B estimates 4/4, and the true Jaccard is 4/5. A against C estimates 1/4, and the true Jaccard is 2/6. After the six steps the counts are 6, 5, 4, 4, 3, and 2. The shipped ids are R5 and R7. Step 1 writes 3 credit lines. The shipped file shares 0 four-word spans with the test. Three synthetic rows stay: S1, S2, and S5.
Check questions
For Job R, how many tokens go into the window, and how many handbook tokens stay on disk? How many windows do the 2,000 dialogues of Job F fill?
Job R puts 1,640 tokens in the window and leaves 78,400 tokens on disk. Job F fills 50 windows.
What is the fingerprint of The cat sat on the mat, and which row id drops because the normalized strings match?
The fingerprint is 96022. R2 drops.
What is row A's four-number signature, what estimate does row B get against it, and what is the true Jaccard of A and B?
The signature is 2, 0, 2, 2. The estimate is 4/4. The true Jaccard is 4/5.
Which two row ids ship in the train file, how many credit lines does the license step write, and how many four-word spans does that shipped file share with the sealed test?
R5 and R7 ship. The license step writes 3 credit lines. The shipped file shares 0 four-word spans with the sealed test.
Put changing text that fits into the prompt, retrieve passages when the source overflows the window, and fine-tune a stable label once you have at least 2,000 checked rows. The shipped file is what remains after the license, the exact string, the near copy, the scrub, the ChatML checks, and a train drop that leaves the sealed test with zero shared four-word spans.
Final task
Submit a data curation pipeline (curate.py) tested against a seeded 5,000-row messy JSONL fixture containing duplicates, broken schemas, PII, and leaked test items; automated tests verify 100% clean output splits and zero train/test leakage.
Terms in this course
This course has no terms of its own in the glossary yet. Terms from earlier courses are used here too.
The first lesson is open to read. The rest of the course is being written, and tests and diplomas come later. Join the waitlist