Skip to content
POLARISAI ACADEMY
R07-401 · Navigator

Sandboxed Agents, Trace Evaluation and Cost-Per-Task

You can normalize a path against a task root, mark each browser and desktop action against the wall rules, score an eight-task fixture for success rate and cost per completed task, and check that a trace root's cost equals the sum of its step spans.

Level
Navigator
Time
28 h (estimate)
Prerequisites
R07-301, R04-301
Course 4 of 4
Jump to

After this course you can

The student can run a sandboxed agent against a 50-task benchmark fixture, emit complete step traces and cost-per-task metrics, and pass all sandbox containment and budget-enforcement tests.

The task root and the path check

A code-executing agent in this lesson runs in a sandbox, one container per task. The container has its own user, a read-only system image, and one writable directory, /work/task-<id>. The host home directory stays outside the sandbox. Host credential files stay outside. The container control socket stays outside. A process that can open that socket can create another container beside the task, so the socket never appears inside the task root. The clock limit for the container is 10 seconds of sandbox time. When the clock reaches 10 the runner stops the container and records the seconds as 10. You can apply the same three outsides on paper today: home, credentials, control socket. If a mount list names one of them, the sandbox line fails before any task result is scored.

Every file call is checked after normalization, before the call runs. Split the path on slashes. Drop empty pieces and the piece .. A piece .. removes the previous real piece. When .. has nothing left to remove, the path stays at /. The call is allowed only when the normalized path is exactly /work/task-<id> or begins with /work/task-<id>/. The raw string is stored on the span so you can redo the steps on paper. Model text is only a proposed path. The mark comes from the normalized form. Network calls use a host list. This fixture allows only files.example.test on port 443. Any other host is denied and recorded. Any other port on files.example.test is denied and recorded. The address 169.254.169.254 is denied on every port. That address is the usual link-local place where a guest asks the host for credentials, so the key marks it deny even when the prompt asks for a metadata service. Containment for the task passes when the runner's allow or deny mark matches the key on every path row and every network row. One row where the key says deny and the runner let the call through is enough to fail containment.

The browser profile and the desktop wall

Browser actions and desktop actions use two different walls, and both walls write rows you can score the same day. The browser tool opens one fresh profile. Its download directory is /work/task-<id>/downloads. Its cookie jar starts empty. The allowed action names are snapshot, click, type, and scroll. Those four are allowed only while the page host is on that task's host list. A navigation to any other host is an egress row and uses the same match rule as a network call from code. A download whose target directory differs from that downloads folder is deny. A file URL is deny. Write three columns for each browser row: action name, host or target path, and your mark. The runner log fills a fourth column. The wall holds when your mark and the log agree on every row.

Computer use drives a pointer, a keyboard, and a screen. When that screen is the host desktop, a key sequence can open a file window or a terminal on the host, and the code container's mounts leave those pixels untouched. This lesson runs desktop actions only on a virtual desktop whose files are the same task root. Allowed rows are pointer moves, clicks, and typing into the focused window of that desktop, plus a screenshot of that desktop. Denied rows are a shortcut or a click aimed at a window outside the virtual desktop, a paste that reads or writes the host clipboard, and a file dialog that can browse above the task root. Each action becomes one span. tool.name is browser or desktop. The span also stores the host or the window title the action touched. You score the wall by writing allow or deny for each row from these rules, then comparing your column with the runner log. The columns match on every row, or the wall failed. A single mismatch is one containment failure, and the loop class later in the lesson reads that mismatch as egress.

Success rate and cost per completed task

An end-to-end fixture is a frozen list of tasks with a written expected result. The arithmetic is the same for 8 tasks and for 50. Each task starts in a fresh sandbox. You record pass or fail, input tokens, output tokens, sandbox seconds, browser or desktop actions, and the containment marks. A pass needs both the expected result and a clean containment column. A correct file reached through an escaped path stays a fail. The price card is part of the fixture, so the units stay fixed from run to run. Each input token costs 2 units. Each output token costs 10 units. Each sandbox second costs 100 units. Each browser or desktop action costs 50 units. Task cost is in×2 + out×10 + seconds×100 + actions×50. Keep the four products in a column so a later reader can add them again without guessing a vendor price.

The success rate is the number of passes divided by the number of tasks. Cost per completed task is the sum of task costs over the whole fixture, passes and fails together, divided by the number of passes. Failed runs stay in the numerator because that spend happened on the way to the passes you keep. The mean cost among passes only divides the passing costs by the pass count. That second figure answers how cheap the successes were by themselves. A report that prints the mean under the heading cost per completed task drops the failed spend. The per-task budget on this fixture is 2500 units. A runner that stops at the budget records the units spent until the stop, the task fails, and those units still enter the numerator. On a 50-task suite you use the same four lines: count the passes, add every task cost, divide that sum by the pass count, and list every task whose cost is over 2500.

Step spans and the loop-class order

The runner writes traces in the OpenTelemetry shape. One trace has one trace id. The task is the root span. Each model call and each tool call is a child span with its own span id, the parent's span id, a name, a start, an end, and attributes. The attribute names in this fixture are task.id, step.index, tool.name, tokens.in, tokens.out, sandbox.seconds, actions, cost.units, containment, and arg.digest. A trace is complete when three checks hold. The child step indexes are exactly the integers 1 through N, with no gap and no duplicate. Every child carries the same task.id as the root. The root cost.units equals the sum of the child cost.units values. A gap, a duplicate index, a wrong task id, or a sum that misses the root fails completeness. The answer file can still look right. Completeness is its own pass, scored beside the task result.

A failed task receives one loop class. Walk this order and stop at the first match. Egress: any path, host, or desktop action was allowed where the key says deny. Timeout: sandbox seconds are 10 or more. Repeat: two steps share the same tool.name and the same arg.digest. Budget: cost.units is greater than 2500. Tool error: a tool span ends with status ERROR and no later child ends with status OK. A pass has the class none. The order is the whole rule when two lines could both be true. You record the first match and leave the later lines unused. You can apply the five lines to a log from a chat export today: one row per step, a key column, and a class written in the margin. On a 50-task suite the same order fills one class cell per failed row, and the counts of those five classes are the loop-failure tally.

An example

Eight tasks, five paths, one trace

Price every task with in×2 + out×10 + seconds×100 + actions×50. T1 passes: in 200, out 40, seconds 2, actions 0, so 400 + 400 + 200 = 1000. T2 passes: in 250, out 50, seconds 3, actions 1, so 500 + 500 + 300 + 50 = 1350. T3 fails: in 800, out 100, seconds 6, actions 0, so 1600 + 1000 + 600 = 3200. Steps 3 and 4 of T3 both have tool.name code and arg.digest d4. T3 has no deny that the runner allowed. T4 passes: in 180, out 30, seconds 1, actions 0, so 360 + 300 + 100 = 760. T5 fails: in 400, out 60, seconds 2, actions 3, so 800 + 600 + 200 + 150 = 1750. The runner allowed the path /work/task-5/../task-1/note.txt. Normalized, that path is /work/task-1/note.txt, and the key says deny because the root is /work/task-5. T6 passes: in 220, out 40, seconds 2, actions 0, so 440 + 400 + 200 = 1040. T7 fails: in 100, out 10, seconds 10, actions 0, so 200 + 100 + 1000 = 1300. T7 has no egress mismatch and no repeated digest. T8 fails: in 300, out 80, seconds 4, actions 1, so 600 + 800 + 400 + 50 = 1850. T8's only tool span ends with status ERROR, and the run has no later child with status OK. T8 has no egress mismatch and no repeated digest.

Add the eight costs from the left. 1000 + 1350 = 2350. 2350 + 3200 = 5550. 5550 + 760 = 6310. 6310 + 1750 = 8060. 8060 + 1040 = 9100. 9100 + 1300 = 10400. 10400 + 1850 = 12250. The passes are T1, T2, T4, and T6, which is 4 passes out of 8 tasks. The success rate is 4/8 = 0.50. Cost per completed task is 12250/4 = 3062.5 units. The four passing costs are 1000, 1350, 760, and 1040. Their sum is 4150, and 4150/4 = 1037.5 units, the mean among passes only. The budget line is 2500. T3 at 3200 is the only task over that line. T5 at 1750, T7 at 1300, and T8 at 1850 stay under it. A 50-task sheet uses this same running total. You still divide the full sum by the pass count.

Classes follow the order egress, timeout, repeat, budget, tool error. T5 matches egress first, because the normalized path left /work/task-5 and the runner allowed it. T7 has seconds 10, so it matches timeout before any later line. T3 matches repeat, because steps 3 and 4 share tool.name code and arg.digest d4, and the budget line is later, so the class stays repeat. If those two steps no longer shared digest d4, T3 would pass the repeat line and match budget, because 3200 is greater than 2500 and the seconds are 6, which sits under the timeout line of 10. T8 matches tool error. The four passes have class none. The tally on this fixture is one egress, one timeout, one repeat, zero budget, and one tool error, even though T3 also sits over 2500.

Normalize the five file calls for task-2, whose root is /work/task-2. Call 1, /work/task-2/out.txt, is already normal and is allowed. Call 2, /work/task-2/../task-5/secret.txt, becomes /work/task-5/secret.txt and is denied. Call 3, /work/task-2/./notes/a.txt, becomes /work/task-2/notes/a.txt and is allowed. Call 4, /etc/passwd, stays /etc/passwd and is denied. Call 5, /work/task-2/../../etc/hostname, climbs to /etc/hostname and is denied. That is 2 allows and 3 denies. A runner that allows call 2 fails containment on that one row, and the class for that task becomes egress. T2's four child costs are 400, 200, 600, and 150. The sum is 400 + 200 + 600 + 150 = 1350, and the root cost.units has to be 1350. The step indexes are 1, 2, 3, and 4. A root printed as 1300 fails completeness, short by 50 units. The split behind those four costs is a model step of 100 input and 20 output tokens (400), a code step of 2 sandbox seconds (200), a model step of 150 input and 30 output tokens (600), and a browser step of 1 second plus 1 action (150). Token totals are 250 in and 50 out, with 3 seconds and 1 action, which is the T2 row above.

Exercise

Four tasks on paper

Use the same price card, the same budget of 2500, the same timeout at 10 seconds, and the same class order. E1 passes: in 100, out 20, seconds 1, actions 0. E2 fails: in 150, out 10, seconds 2, actions 2, and the runner allowed a navigation to other.example.test while the key says deny. E3 passes: in 80, out 10, seconds 2, actions 0. E4 fails: in 50, out 10, seconds 10, actions 0, with no egress mismatch and no repeated digest. Write each cost, the success rate, the cost per completed task, the mean among passes only, and the class of E2 and of E4. Then normalize /work/task-e1/../task-e2/x.txt and /work/task-e1/a/../b.txt against the root /work/task-e1, and mark allow or deny. A trace for E1 prints child costs 200, 150, and 150 with a root cost of 480. Say whether that trace is complete, and by how many units the root misses the child sum. Name the class T3 would get if digest d4 were not repeated.

E1 costs 200 + 200 + 100 = 500. E2 costs 300 + 100 + 200 + 100 = 700. E3 costs 160 + 100 + 200 = 460. E4 costs 100 + 100 + 1000 = 1200. The four costs add as 500 + 700 = 1200, then 1200 + 460 = 1660, then 1660 + 1200 = 2860. Passes are E1 and E3, so 2 of 4, and the success rate is 0.50. Cost per completed task is 2860/2 = 1430 units. The passing costs sum to 500 + 460 = 960, and the mean among passes only is 960/2 = 480 units. No exercise task is over 2500. E2 is egress. E4 is timeout. The first path normalizes to /work/task-e2/x.txt and is deny. The second normalizes to /work/task-e1/b.txt and is allow. The child sum is 200 + 150 + 150 = 500. The root says 480, so the trace fails completeness and the root is short by 20 units. Without the repeated digest, T3's class is budget.

Check questions

Answer on your own first. Then open a question to see the answer.

After normalization, what path is call 2 for task-2, and is that path allowed for root /work/task-2?

/work/task-5/secret.txt, and it is denied. The raw call was /work/task-2/../task-5/secret.txt, and the normalized path leaves the task root.

On the eight-task fixture, how many tasks pass, what is the success rate, what is the cost per completed task, and what is the mean among passes only?

4 passes, success rate 0.50, cost per completed task 3062.5 units, and the mean among passes only is 1037.5 units.

Which three checks must hold for a task trace to be complete?

Child step indexes are exactly 1 through N with no gap and no duplicate, every child has the root's task.id, and the root cost.units equals the sum of the child cost.units values.

Which loop class does T3 receive, and which class would it receive if steps 3 and 4 no longer shared arg.digest d4?

T3 receives repeat. Without that shared digest the class is budget, because 3200 is greater than 2500 and the sandbox seconds are 6, under the timeout line of 10.

Take with you

The sandbox scores every path, host, and desktop action against one task root and one allow list. Success rate is passes over tasks. Cost per completed task puts every run, including the failures, into the numerator. A trace is ready to grade when the child costs add up to the root and the loop class is the first rule that matches.

Final task

Submit a sandboxed agent runner and trace evaluator that execute against a frozen 50-task suite in the test container, passing all containment checks, staying within the per-task token budget, and producing exact keyed trace metrics.

The final task is marked against fixed answer keys and automated tests once the tests open.

Terms in this course

This course has no terms of its own in the glossary yet. Terms from earlier courses are used here too.

More lessons coming

The first lesson is open to read. The rest of the course is being written, and tests and diplomas come later. Join the waitlist