Personal Agent Bench

How to evaluate AI agents: how our baseline tasks were drawn

Most guides on how to evaluate AI agents start with metrics and judges, but an evaluation can decide its results without changing a single grading rule, simply by choosing which tasks to run. Pick the errands one product is good at and it wins. Pick the ones another product trips on and it loses. So this page publishes the rule that chose the tasks for our baseline batch, before any results from it exist, along with what the rule can and can't protect against. If you are evaluating agents yourself, the lesson carries over directly: write down how tasks are chosen, and fix the list, before you look at a single result. Anthropic's engineering guide to agent evals tells teams to test both the cases where a behaviour should occur and where it shouldn't, because "one-sided evals create one-sided optimization". This page is what that looks like when the balance has to be decided by a rule rather than by taste.

  • The historical note reports a draw from 312 tasks with fairness exclusions. The historical excluded IDs have not been reconciled against the five currently listed tasks.
  • Tasks were grouped by what the agent actually has to do, and no group contributes more than two tasks. Nobody wins by being good at one errand repeated thirty times.
  • The historical note reports a seeded draw on 4 October 2026 independent of product results; the inputs and exclusions needed to verify that independence are not public.
  • The list holds 169 tasks, each to be run twice for every product. None of the six October batches published so far is this draw: they used 7 to 72 tasks each, and the material we hold does not say how those were picked.
  • Every limit of the draw is listed below, including the worlds it ended up weighting most heavily.

Why task selection decides how you evaluate AI agents

The scorecard's rule has always been to publish how something is measured before publishing the measurement. Grading rules are the obvious case, and they are on how the exam works. Task selection is the less obvious one, and it is at least as powerful.

Every task in the library is a legitimate errand. That's what makes selection dangerous: a set chosen to flatter one product doesn't contain a single bad task, so nobody looking at it can point to the trick. The only defence is to fix the rule in advance, apply it mechanically, and say what it was. If the results then favour someone, they favour someone under a rule that was written down before anyone knew which way it would cut.

Not everyone selects this way, and the alternatives are worth knowing. A March 2026 paper, Efficient Benchmarking of AI Agents, proposes running new agents only on tasks whose historical pass rates sit between 30% and 70%, and reports that this cuts the task count by 44 to 70% while keeping rankings stable. That is a sensible tool for ranking many agents cheaply. It also reads past results to choose tasks, which is exactly what our rule refuses to do, because we publish a rate per product rather than a ranking.

How to evaluate AI agents in five steps

  1. Define success from the final state of the world, not from the agent's reply. How the exam works shows that rule in practice.
  2. Fix the task selection rule before looking at any result. That is this page.
  3. Repeat each task and print the uncertainty interval beside every rate. The section below on what the draw can support explains why one run per task is not enough.
  4. Check the grader, including whether a product can pass by doing nothing.
  5. Keep goal completion, reply quality and unwanted actions as separate numbers. The recorded runs show all three for every attempt.

Two planned runs per task are this protocol's choice, not evidence that two trials settle reliability.

Start from all the tasks, then remove what can't be run fairly

The library held 312 tasks when the draw was made. Some of them run against a real outside account, and those can't be reset to the same starting point for every product; a task that starts differently for each product isn't the same task, so they were excluded.

The count of those exclusions has a loose end. The historical draw note records three real-account exclusions. The current library has five such tasks: Google Calendar self-event, Notion self-page, self-only calendar read, self-only calendar write, and Slack self-message. The draw record does not identify which three were excluded, so we cannot reconcile the older count with the current list. The 312-to-309 arithmetic below describes that historical draw, not a fresh exclusion of all five.

One more was removed after the draw: the task that has to get past a page checking for automated browsers. The browser the exam supplies can't disguise that it's automated, so every product using it would fail that task for a reason that belongs to us. The browser tasks page covers that task and the four browser tasks that stayed in.

The historical draw records no other removals. The draw note says it did not select on product results; without the full historical inputs we cannot independently verify that account.

Group tasks by what the agent has to do

The library has many near-relatives. One booking task asks for a table for four, another for a table for two on a different night, a third adds an allergy. They exercise the same skill against slightly different facts. If they were drawn freely, whichever skill has the most variations would dominate the batch, and a product strong at that one skill would look strong overall.

So before drawing, every task was placed in a group by what the agent actually has to do. Two tasks are in the same group when they run in the same world and the agent's standard solution makes the same set of changes to it. Reads don't count toward this, because nearly every task starts by reading something. What defines the group is what the agent is supposed to change.

Tasks graded only on the agent's reply, with no changes to make, were grouped by world and then split further by what they need from the agent, such as remembering something across sessions, and by which faults they inject. Without that split they would all collapse into one huge group, and memory tasks in particular would crowd each other out.

The 309 remaining tasks (the robot-check task was still among them at this stage) fell into 128 groups.

At most two tasks per group

From each group, at most two tasks were taken. A group with only one task contributes that one. A group with forty contributes two.

The cost of this is deliberate. A skill that the library tests forty different ways now counts for as much in the batch as a skill it tests twice. That under-weights whatever the library happens to have the most of, which is exactly the point: how many variations someone wrote for a skill says more about the authors than about how often you'll need it.

Two rather than one gives each group a second task with different facts, so a product doesn't pass a whole skill on the strength of one lucky reading. Raising the limit to three was considered as a way to grow the sample later, and it is recorded as an option rather than the current rule.

The second pick fills a gap

Within a group, the first task is simply the first one after a seeded shuffle. The second is chosen to cover something the first didn't.

The exam tags tasks with the kinds of difficulty they carry: graded only on the reply, has an injected fault, uses a connector, requires the agent to act without being asked, spans more than one session, needs answers from the simulated user, needs approval, or tests a safety boundary. If the first task in a group lacks one of these and another task in the group has it, that task is taken second. If no task fills a gap, the second task is again just the next in the shuffle.

The effect is that the two tasks drawn from a group are as different as the group allows, without anyone choosing them by hand.

The reported fixed seed and selection independence

The shuffle used a single fixed seed, the date of the draw written as a number. The seed, the grouping rule and the gap-filling rule are written at the top of the task list itself, but the full task list and selected IDs are not public, so readers cannot reproduce the draw from this page.

The described rule does not use product results as an input. The historical note reports that this rule governed the draw, but without the historical inputs and exclusions we cannot verify that independence in execution. Publishing a mechanical rule makes the intended selection process explicit; it does not by itself establish what happened in the historical draw.

What the draw looks like

The list holds 169 tasks. All seven errand worlds and three of the four small browser worlds are represented; the fourth is the robot-check world left out above. The distribution is uneven:

WorldTasks drawn
Work files and documents67
Email and calendar30
Bookings and tickets20
Life admin and bills15
Shopping and after-sales15
Travel13
Live data streams5
Browser skills, across three small worlds4

Work files make up 40% of the draw, and that follows directly from the rule. The cap is per group, not per world, so a world with many distinct kinds of change gets many groups and therefore many tasks. Work file tasks involve many distinct kinds of change. Live data tasks involve few. We could have forced the worlds into equal shares, and that would have meant overriding the mechanical rule with a judgement about which worlds matter, which is the thing the rule exists to avoid. Instead the shares are printed here so you can discount them yourself. A product strong at work files will look better overall in this batch than in one balanced by world, and that should be read as a property of the draw.

What this draw can and can't support when evaluating AI agents

Each task will be run twice for every product. Anthropic’s evaluation guide distinguishes success at least once across k trials (pass@k) from success on every one of them (pass^k), and gives no fixed trial count. Two is this protocol's chosen repeat count, not a general reliability standard, and the published record does not give a reason for choosing two rather than more.

The October batches show what one run per task looks like. Each product answered each task once, so on the 11 October headline batch a single attempt more or less moves a rate by five to seven points, and Instinct's 40% sits inside an interval from 20 to 64%. Three batches ran on 11 October from the same 20-task list, with exam code and product builds changing between them, and Muse came out at 37%, 58% and 33%. Those three are never pooled, and the gap between them is not only luck. It is still a fair picture of how far one pass over a small task set can swing. Two attempts at 169 tasks gives several hundred attempts per product, but attempt count alone does not fix the precision: that depends on the outcomes and on an estimator that accounts for repeated and grouped tasks, which has not been published here.

It also can't tell you about any one task. Two attempts at a single errand is an anecdote, and per-task results are shown as what happened, never as a rate; the run pages count records for exactly that reason. And it can't tell you about errands the library doesn't contain. The groups cover what the exam knows how to test, which is a subset of what you'll actually ask an agent to do.

How this task selection relates to the published results now

Two sets of results are published on the results page, and neither came from this draw. The September rates were measured on 22 September, before this rule existed, on a different set of tasks under the earlier grading rules. The six October batches ran between 11 and 14 October under the new rules, on 7 to 72 tasks each; 59 distinct tasks appear in their published records. Neither is a smaller version of this batch, and neither should be read as a preview of it.

When the batch drawn by this rule reports, it will start its own series. It will be shown beside the others with its date, task set and rules, and never joined to them in a single line. A product that moves between them may have changed, or the tasks may have, or the rules may have, and a chart that joined them would hide which. Keeping them apart is less satisfying to look at and is the only version that doesn't mislead.

FAQ

Could the draw still favour a product by accident? Yes, and the world shares above are the clearest way it could. The draw algorithm does not read results; library construction, grouping and historical independence remain unverified. It doesn't guarantee the result is neutral between products with different strengths, and no selection rule could.

Why not run every task? Because the library is uneven in the way described above, and running all of it would make the batch a measure of which skills have the most variations. Running every task twice for every product would also cost nearly twice as much, for a result that's more lopsided rather than more accurate.

Will the draw change? When it does, the new rule will be published before its results, and the change will be dated on what changed in the exam.

Can I see which tasks were drawn? The list itself isn't published on this site. What's published here is the rule, the counts and the shares. The errand pages describe the kinds of work the tasks cover, and the run pages show the tasks the October batches used, attempt by attempt.

Where the task draw sits in our evaluation

The grading rules are on how the exam works, how the exam's own judgements are checked is on how we check our own grading, and every change to either, including this draw, is dated on what changed in the exam.