Chatbots & assistants
Compare chatbots by hiding the names and keeping the reply that does the job. Then use public boards for preference, invented facts, and price, and keep your own sheet until a fixed Swedish set is published.
Beginner
Hide the names, then choose
A chatbot lets you write to a large language model, a system that continues your text with a likely reply. An assistant is that model in a product that can also search, open a file you upload, or follow instructions you saved. Test either one the same way: give two products one task, hide the names, and keep the reply you would actually send.
Paste the prompt below into two chats. Cover the logos. Choose the email you would send, then look at the names and write one reason, such as a clearer request, a better tone, or fewer invented details. An invented detail is a hallucination. The model stated something as fact, your message did not contain it, and a check shows it is wrong.
Use three jobs you can already judge. An email should ask for one action and stay polite. An explanation should teach one idea and define any new word in the same sentence. A translation should keep names, dates, and amounts, and should say when a phrase has no close match. If the answer depends on a live fact, read the official page for that fact before you act.
Arena's Text leaderboard collects blind picks like these and turns the wins into a score. A higher score means voters preferred that model's replies in the recorded matchups. For health, law, or money, check the source yourself. Discuss this page in the forum. During the trial it is open to invited members. Save the prompt, your winner, and the reason.
Dripping tap email
Write a polite email to my landlord. The kitchen tap has been dripping for a week. Ask for a repair visit during this coming week. Stay under 120 words. Use a polite greeting and a clear request in the last sentence. After the email, list every fact you used that I did not give you.
Intermediate
Same five prompts, written down
Write five prompts and do not edit them during the run: an email, an explanation, a translation into Swedish, an everyday question you can check, and a question where the honest answer admits a missing fact. Send each prompt to each model on the same day. Record the product, the model name if shown, the date, and any visible switch, such as fast mode or a thinking mode. If one product searches the web, turn search on for the rest or mark that row as a separate test.
Score each reply 0, 1, or 2 on four lines: it did the task, it stayed within the length you asked for, it added no unsupported fact, and the tone matches. Keep the total and the full text.
Arena's Text board ranks models from blind human votes. Style control on that page lowers the advantage of a long or heavily formatted reply. Artificial Analysis compares an intelligence index, price, output speed, wait before the first token, and context window. A token is a piece of text, often a word or part of a word. The context window is the most tokens the model accepts in one turn, and the reply limit is often lower.
For Swedish, add a letter you might send to a landlord, a school, or a municipality, and ask for plain wording. Mark lines that sound translated, that mix formal and casual address, or that split a Swedish compound word in two. Repeat the five prompts after a model change and keep the old rows.
Plain Swedish repair letter
Write a letter in Swedish to a municipal housing office. The lamp in the stairwell outside my flat has been out for ten days. Ask them to repair it, and offer two weekday times next week when I am home. Use plain Swedish of the kind a caseworker expects in a written request. Stay under 140 words. After the letter, add an English note that marks any sentence which may sound translated, and say why.
Expert
Rows another person can rerun
Record a row another person can rerun: model version, date, settings, prompt id, score, and price. Arena turns blind pairwise votes into a Bradley-Terry score, in the same family as an Elo chess rating. The Text board shows that score, a confidence range, a vote count, and a rank spread. Style control gives less credit to a win that comes from length or formatting. Dated snapshots of the text, vision, and agent boards are in the lmarena-ai leaderboard dataset.
Vectara asks the model to summarize a private document from that document only, at temperature 0 when the API allows it, and scores the summary with HHEM-2.3, its Hallucination Evaluation Model. GSM8K is 8.5 thousand grade-school word problems with the final number marked. LiveCodeBench adds new contest tasks in numbered releases and reports pass@k, the chance that k samples include one program which passes the tests.
Cost is input tokens times the input price plus output tokens times the output price, with both prices given per million tokens. Artificial Analysis publishes price, speed, context length, and a long-context test. Its Data API returns the same fields.
For safety, follow the provider's published rules and the OWASP Top 10 for applications built on large language models. A jailbreak is a message written to push the model past those rules. Record refused, partial, or complied for each banned category, and publish only the category and the result. A system prompt is the instruction the product adds before the user message.
A fixed Swedish set on plain-language law, local humor, and admin letters is not published yet.
Score the reply against the source
Score one chatbot reply for factual consistency. I will paste a SOURCE and an ANSWER. Use only the SOURCE. Do not add facts from your own knowledge. Return three counts: claims the SOURCE supports, claims absent from the SOURCE, and claims that contradict the SOURCE. Quote each absent or contradicting claim. Then label the reply as refused, answered, or mixed. If I have not pasted a SOURCE, ask for it and stop.
A sheet for every reply
Copy this row once per answer.
- Date and product name
- Model version, copied from the screen
- Search on or off, and fast or thinking if the product offers it
- Prompt id and the full reply
- Scores for task, length, unsupported facts, and tone
- Token counts if shown, including reasoning tokens and cache writes, plus the price
- Whether you would send or rely on the reply
Which public test answers which question
| Question | Open this |
|---|---|
| Which reply did people prefer with the names hidden? | Arena Text |
| Which reply was preferred on a picture? | Arena Vision |
| Did a tool-using session finish, and what did it cost? | Arena Agent |
| How often did a summary add facts missing from the source? | Vectara hallucination leaderboard |
| What are price, speed, and context window? | Artificial Analysis |
| Can a script mark grade-school maths? | GSM8K |
| Can a script mark newer contest code? | LiveCodeBench |
Copy the date on the board into your sheet. Ranks move when new votes arrive.
What this page will add
The blind comparison screen and the fixed Swedish prompt set are still to be built. A finished run will store the prompt id, both full replies, the vote, the model names after the vote, the date, and the machine name. Until those tools exist, the sheet on this page is the method.
A shared community DGX Station would run the same Swedish prompt set on large open-weight models that do not fit on a laptop, and each run would be filed with the date and the machine name. About the shared station
Sources
- Arena Text leaderboardBlind votes on text replies, plus style control and category filters. Picture replies are at https://arena.ai/leaderboard/chat/vision and tool-using sessions at https://arena.ai/leaderboard/agent.
- Arena leaderboard datasetDated snapshots for text, vision, and agent boards, with a latest split and full history. The card records the change from Elo to Bradley-Terry and when style control became the default.
- Vectara hallucination leaderboardMethod and table for how often a summary adds facts absent from the source, using HHEM-2.3 at temperature 0. The README links the interactive board at https://huggingface.co/spaces/vectara/leaderboard.
- Artificial Analysis model comparisonSide-by-side intelligence index, price, speed, latency, and context window. The same fields are documented for scripts at https://artificialanalysis.ai/data-api.
- GSM8K grade-school math8.5 thousand human-written word problems with the final number marked. The repository was archived on 8 April 2026 and is read-only.
- LiveCodeBenchContest tasks gathered over time from LeetCode, AtCoder, and Codeforces, with numbered releases and pass@1 and pass@5.
- OWASP Top 10 for Large Language Model ApplicationsProject page for OWASP's list of the main security risks in applications that use large language models, with the link to the current list.