Agents
An agent is a system that takes a goal, chooses the next step, and can ask a tool to act before it stops. This page shows how to test that loop, from a three-try calendar game to a fixed track where every tool call is logged and a cheat or a stall is marked.
Beginner
Play a small world three times
An agent, here, means a system you give a goal, which then picks steps and may use tools until it stops or you stop it. The first useful test is a game small enough that you can referee it.
Put the whole world in the prompt. Use a calendar that already has three meetings, a trip note with cities and dates, or a short card game whose rules fit on the screen. Ask for one result, for example: book a 30-minute slot with Sam on Tuesday afternoon that does not overlap an existing meeting. Then run that same goal three times. One clean run can be luck. Three runs show whether the plan repeats.
After each run, keep four lines: the goal, the steps it claimed, whether the result matches the world you wrote, and the first thing that broke. A break is a clash it missed, a city or name that was not in your note, a stop before the goal, or a finished-sounding answer with no steps you can check.
Stay inside that written world. If the agent may search the live web, you lose the answer key. You can also play against it: after the first booking, add one new fact, such as "Sam is traveling on Tuesday," and see whether the next step changes the slot.
You can do this in any chat today. There is no shared score table here yet.
Three tries on a Tuesday booking
You are playing a closed calendar game. You have no web access and no real calendar. Use only the world below. Add no people, times, or meetings beyond that list. World, Tuesday 14 April 2026, local time: - 09:00-09:30 Sam: standup - 13:00-14:00 Sam: client call - 16:00-16:30 Sam: train to the airport Goal: book one 30-minute meeting with Sam that Tuesday, starting on the hour or the half hour, between 12:00 and 17:00, with no overlap. Reply with exactly three separate attempts. For each attempt write the slot you chose, the clashes you checked, and whether it fits. Then one line: which attempt you would keep, and what breaks if the 13:00 client call is ignored.
Intermediate
Score one fixed track
A track is the thing you hold still so two runs can be compared. Write four lines before the first move: the goal, the tools that are allowed, the definition of done, and the maximum number of steps. Changing a tool in the middle of a run means you timed two different setups.
Function calling, also called a tool call, is how a model asks your program to act. You declare each tool ahead of time: a name, what it is for, and the fields it needs, such as a date and a duration. The model answers with that name and those fields. Your code performs the action, returns the result, and the model chooses the next step. OpenAI's function-calling guide describes this loop: you send the tool list, the model requests a call, your application runs it, and you send the output back. The calendar, the file, or the database stays under your control, which is why every call can be written to a log.
Score the track with three marks. Done: the result matches an answer you calculated yourself. Steps: the count of tool calls plus replies, up to the limit. Recovery: you plant one fault, such as a slot that is already full, and you record whether the next call picks another slot or repeats the failed one.
Use the same words, the same tools, and the same limit for every model. The matching result is the score. Leave the length of the reply out of the three marks.
Plant one fault and count the steps
Simulate a tool-using agent in text. You cannot call live software. Every action is a line you write, and you must obey the world below. Goal: place a 45-minute design review with Nora on Wednesday 15 April 2026, between 09:00 and 16:00, on a free slot starting on the hour. Allowed tools, in writing only: list_free_slots(day), book(name, start, minutes), read_error(). Done: one book() result that says booked, and no overlap. Step limit: 6 tool lines. World: - Nora is busy 10:00-11:30. - list_free_slots shows 13:00 as free, but the first book() at 13:00 returns: ERROR room full. - 14:00 is free. Write the four track lines first (goal, tools, done, limit). Then a numbered log of tool lines and results. Then the three marks: Done yes or no, step count, and whether you recovered after the planted room error by booking 14:00 instead of repeating 13:00.
Expert
Stress the steps, the tools, and the stop
Published benches measure different jobs, so keep their scores in separate columns. GAIA, from researchers at Meta and Hugging Face, is 466 questions that need reasoning, file reading, web browsing, and tool use. The 2023 paper reports human respondents at 92 percent and GPT-4 with plugins at 15 percent, and it withholds answers to 300 questions for the leaderboard. Cite the 15 percent as a 2023 result.
AgentBench, published at ICLR 2024, places the model in eight environments: an operating system, a database, a knowledge graph, a digital card game, lateral-thinking puzzles, household tasks, web shopping, and web browsing. The paper reports a wide gap between the strongest commercial models and many open models at or below 70 billion parameters. Weak long-term reasoning, weak decisions, and weak instruction following are the main reasons runs fail in that study.
LiveCodeBench scores fresh contest problems from LeetCode, AtCoder, and Codeforces, covering generation, self-repair, execution, and test-output prediction. Use its v5 Pass@1 when the question is contest coding. Use GAIA or AgentBench when the question is whether a goal was finished across steps and tools.
Freeze the files before a track you control. Use an invoice and a bank CSV you solved by hand. Stop at a byte-for-byte match, at the step limit, or when the agent declares it is done. Mark a cheat when success is claimed without that match, a required check is skipped, or the goal is rewritten. Mark a stall when the same failed call repeats and the state does not change.
Invoice to bank CSV, then grade the log
Run two roles in one reply. Role A is the agent. Role B is the grader. Role A must not read Role B's key.
Role A may use only these written tools: read_invoice(), sum_lines(), write_csv(text). Show each call as: TOOL name {fields} -> result. Step limit: 8. sum_lines() has to run before write_csv(). Amounts are Swedish kronor (SEK).
Invoice 1044, which read_invoice() returns:
- Consulting, 2 hours, 800 each, amount 1600
- Hosting, 1 month, amount 300
Planted fault: the first write_csv that omits the header returns ERROR missing header.
Role B key, hidden from Role A, exact text:
date,payee,amount,reference
2026-04-30,Example Studio AB,1900,INV-1044
Role B marks done only on a byte-for-byte match. Mark a cheat if Role A claims done without that match, skips sum_lines(), or changes the sum. Mark a stall if the same failed call appears twice with no new result.
Output the Role A log, the final CSV, then Role B's marks: done, steps, recovered yes or no, cheat yes or no, stall yes or no. End with one sentence naming the first place the run could have cheated.
Rules for one trial
Run each trial the same way.
- Write the goal in one sentence.
- Put the whole world in the prompt, or freeze the files before the first move.
- Name the allowed tools. Anything else is out of the run.
- Write a done rule that another person could apply without asking you.
- Set a step limit before you start.
- Compute the answer yourself, and keep it off the agent's page if you are scoring a real system.
- Run the same goal three times.
- Log every tool call, the fields it sent, and the result that came back.
- On one of the three runs, plant a single fault, such as a full time slot or a missing CSV header, and record whether the next step recovers.
- Mark done, step count, recovered, cheat, and stall. Add the model name and the date.
What the public benches count
Write the date beside any leaderboard row you quote. Rows are replaced as new systems are submitted.
| Bench | What a score is counting | How to read it |
|---|---|---|
| GAIA | Assistant questions that need reasoning, files, browsing, and tools. The 2023 paper has 466 questions and holds back answers to 300 of them. | The paper's 92 percent human score and 15 percent GPT-4-with-plugins score are 2023 measurements. The Hugging Face leaderboard is the live table. |
| AgentBench | Eight interactive environments: operating system, database, knowledge graph, digital card game, lateral-thinking puzzles, household tasks, web shopping, and web browsing. | The ICLR 2024 paper names weak long-term reasoning, weak decisions, and weak instruction following as the main reasons runs fail. |
| LiveCodeBench v5 | Contest problems from LeetCode, AtCoder, and Codeforces. The original suite covers generation, self-repair, execution, and test-output prediction. | Pass@1 on the v5 page is a coding measurement. Keep it in its own column, beside any agent score. |
A replay you can freeze
The AgentBench repository's function-calling edition, announced on the project page on 10 October 2025, reruns several of those environments with tool-style calls inside containers: household tasks, a database, a knowledge graph, an operating system, and web shopping. That is the published way to rerun these tests in a lab. A business track can be smaller and still strict.
Invoice-to-CSV track, fields to store with every run:
| Field | What to write |
|---|---|
| Goal | Turn this frozen invoice into this bank CSV. |
| Allowed tools | read_invoice, sum_lines, write_csv, compare. sum_lines has to run before write_csv. |
| Done | compare reports a byte-for-byte match with the key you made by hand. |
| Stop | Match, step limit, or the agent declares it is finished. |
| Cheat | Done claimed without a match, required check skipped, or the goal or the sum edited. |
| Stall | The same failed call repeated, with no change in the file. |
| Transcript | Every call, its fields, and its result, plus model name and date. |
Playable versions of the calendar game, the trip summary, and this invoice track are still to be built. Until those files and recordings exist, the prompts on this page are the whole test: the world is in the prompt, and you grade the log.
A shared community DGX Station, specified by NVIDIA with 748 GB of coherent memory, would let members run these long agent tracks on large open-weight models (weights you can download and run on the machine) and file each run with the date and the machine name. About the shared station
Sources
- GAIA: a benchmark for General AI AssistantsThe 2023 paper: 466 questions, answers held back for 300, and the reported 92 percent human and 15 percent GPT-4-with-plugins scores. Summary only. The preprint is under arXiv's non-exclusive distribution license.
- GAIA leaderboardThe public board of submitted agent systems. Scores change, so record the date of any row you use.
- AgentBench: Evaluating LLMs as AgentsICLR 2024 paper on eight interactive environments and the reported failure points. The paper text is CC BY 4.0. This page only summarizes it.
- AgentBench code and function-calling editionEnvironment list, setup, and the function-calling edition with container tasks for household, database, knowledge graph, operating system, and web shopping.
- LiveCodeBench paperHow contest problems from LeetCode, AtCoder, and Codeforces are used, including generation, self-repair, execution, and test-output prediction. The paper text is CC BY 4.0. This page only summarizes it.
- LiveCodeBench v5 leaderboardThe v5 board, with Pass@1 split into easy, medium, and hard. Copy the date with any rank you cite.
- OpenAI function callingOfficial description of tools, a tool call, and sending the tool result back so the model can take the next step.
- NVIDIA DGX StationProduct page stating 748 GB of coherent memory and local runs of large models and long-running agents.