Tests
A test gives two or more AI tools the same job, keeps the names hidden until you have judged the replies, and saves the prompt and the answer so the run can be repeated.
Results
No published test results yet. Until then, this page shows how to run your own: five jobs, a score from 1 to 5, and a log row anyone can repeat.
Beginner
Three rules and a first score
A test here means one fixed job, given to an AI chat, with the reply saved. The job can be a letter, a summary, a sum, a source check, or reading text in a picture. You are checking whether the reply does that job.
Use three rules every time. Give every tool the same task, in the same words, with the same files. Hide the names until you have picked a winner, the way a blind comparison works on the public Arena site: you vote, and only then the names appear. Save the prompt, the full reply, the date, and the name of the tool once you reveal it.
Score each reply from 1 to 5 using the table further down, and write one sentence on why.
Start with a letter you actually need. Tell the chat who it is for, what you want, and which facts it must not add. Then read the draft the way you would read a message from a new colleague: names, dates, promises, and tone. If you work in Swedish, run that same letter in Swedish as well. A tool can sound smooth in English and still stumble on Swedish grammar or the wrong level of politeness. Keep the sheet yourself. The five jobs in the checklist are enough for a first evening.
Letter with no invented facts
Write a short email from me to my landlord. I need the hallway light repaired this week. My name is [your name]. The address is [street, postcode, and city]. Do not add a time of day, a phone number, or any previous complaint. Use plain polite language. After the email, list every fact you used, and mark any detail that I did not give you.
Intermediate
Read each public table for the job it measures
Public tables answer different questions, so read the label before you copy a rank into your notes.
Arena collects human votes on anonymous pairs. A person types a prompt, sees two replies with the names hidden, and picks the one they prefer. The rank records which reply voters preferred. Length and tone can move that vote.
Artificial Analysis publishes separate figures for quality-style indexes, price per million tokens, and speed. Match the column to the job. A low price still leaves the letter, the sum, and the Swedish wording for you to score.
SWE-bench asks a model for a code patch that fixes a real GitHub issue, then checks that patch with the project's tests. A strong result covers software repair. Summarizing meeting notes is another job, so keep that score in its own column.
The EleutherAI Language Model Evaluation Harness runs fixed tasks and records a task version, so two runs can name the same setup. Many of those tasks look like English exams. Your five jobs, in the language you use, still belong on the sheet.
Run all five jobs on two tools. Change one thing at a time: the tool, the language, or the wording. Leave temperature at the product default and write that down. Temperature is the setting that makes word choice more varied as the number goes up. Put both replies in a table with their stars and a one-line reason. If you cite a public table, copy the site name, the benchmark name, and the date beside your stars.
Blind score of two replies
I will paste a task and two replies labeled A and B. The tool names are hidden. Score each reply from 1 to 5: 1 misses the job or invents a fact, 3 is usable after edits I can name, 5 I would send or trust with no edits. For each reply, give the stars and one sentence that points to a specific line. Then say which reply you would keep, and what I should still check by hand. Do not guess which product wrote which reply.
Expert
Log a run so someone else can repeat it
A repeatable test writes down every choice that can change the reply.
Record the model name and exact version or web product, the date and time, and the system text when the product shows it. Record the full user prompt and any file or image, with name and size. Record temperature, top-p if you set it, and the seed. A seed is the starting number for the random draws the model uses. The same settings can still differ across machines, so record the provider or machine and the software version. Save the reply as text. When the tool shows it, note time to first token (the wait before the first word) and the time until the reply finishes.
EleutherAI's evaluation harness reports a task version and takes a seed plus settings such as temperature. Store that version next to the score. The code is MIT-licensed: keep the copyright and license notice if you reuse it. SWE-bench, also MIT-licensed, scores a patch by whether the project's tests pass. Name the dataset split and the harness version.
MLPerf, from MLCommons, measures hardware. Training records how long a system takes to train a model to a stated quality target. Datacenter inference records how fast a trained model processes inputs, with latency, throughput, and optional power at the wall. Keep those numbers in their own columns, with the suite version.
Freeze the prompt in a file, name the human scorer, and say whether the run was blind. Read the current Artificial Analysis data-API terms before you republish their figures.
Build the log row, invent nothing
Help me log one AI test so someone else can repeat it. Ask for any field I have not given you. Do not fill gaps with guesses. Fields: date and time with time zone; model name and exact version or product; provider or machine; software version; language; temperature; top-p; seed; system text; full user prompt; attached file names and sizes; time to first token; time to finish; full reply; scorer name; whether the run was blind; stars from 1 to 5; one sentence on why; public site, benchmark name, and date if I cite one. When every field is filled from me, output a single Markdown table with one row.
Five jobs to run this week
Run each job on two tools with the same words. Hide the names until you have scored both.
- Letter. Ask for an email you need this week. Name the reader, the request, and the facts that must appear. Tell the tool not to add a phone number, a time, or a past complaint you did not supply. Score tone, missing facts, and added facts.
- Summary. Paste a text you are allowed to share, such as a page you wrote. Ask for a summary in a set number of sentences. Check which points were dropped, and mark any claim that was not in the source.
- Arithmetic. Write a short receipt or a word problem, and solve it on paper first. Ask the tool to show each step. Compare the final figure with yours.
- Source check. Ask for one factual claim and the page that supports it. Open every link. A dead link, or a page that does not say what the reply claims, lowers the score.
- Text in a picture. Photograph a sign, a menu, or a receipt. Ask for every word and number, in order. Compare the transcript character by character.
Score from 1 to 5
Use the same meaning of each star on every job.
| Stars | What you saw |
|---|---|
| 1 | The reply misses the job, or it states a fact you did not give and that you can show is wrong. |
| 2 | Part of the job is done. You would rewrite most of the reply. |
| 3 | You can use the reply after you fix errors you can name. |
| 4 | You would change a few words, and the facts check out. |
| 5 | You would send the reply, or trust the figure, with no edits. |
Add one sentence that points at a specific line. A smooth tone with a wrong figure stays at 1 or 2.
What to store in each row
Save one row per run in a spreadsheet. During the trial, invited members can also post the rows in the forum.
- Date and time, with the time zone.
- Tool or model name, and the exact version or product screen you used.
- Provider or machine, and the app or software version.
- Language of the prompt.
- Temperature, top-p, and seed, or the words "product default" if the screen does not show them.
- The full prompt and the names and sizes of any files or images.
- The full reply, saved as text.
- Time to first token and total time, if the tool shows them.
- Your name as scorer, whether the names were hidden, the stars, and one sentence on why.
- If you mention a public table: site, benchmark name, and the date you read it.
If Polaris buys a shared station, heavy tests on large open models could be repeated there, with the same settings written on every row. About the shared station
Sources
- How Arena worksHow a public blind comparison works: you submit a prompt, see two unnamed replies, vote, and only then see which models wrote them.
- EleutherAI Language Model Evaluation HarnessA runner for many fixed language-model tasks. It records a task version and accepts a seed and generation settings such as temperature. MIT license (license file).
- SWE-benchA benchmark that asks a model to patch a real GitHub issue and then checks whether the project's tests pass. MIT license (license file).
- MLPerf Inference: DatacenterMLCommons results for how fast server systems run a trained model, with latency and throughput rules and optional power measured at the wall. Closed division keeps the reference model; open division may change it.
- MLPerf TrainingMLCommons results for how long systems take to train a model to a stated quality target. Each benchmark names its dataset and target.
- Artificial Analysis Data APIDocumented access to their published quality indexes, token prices, and speed figures. Read the current terms before you copy any figure into a public table. Charts are on the Artificial Analysis website.
- Hugging Face LLM course, chapter 1Free course on language models and the Transformers library. The page says the course is Apache 2.0, so credit it and the license if you reuse the material. It expects Python.
- Elements of AIFree introductory course from the University of Helsinki and MinnaLearn. Their license page says the lessons are all rights reserved: you may cite them, and you may not copy the text or exercises (license page).