Personal Agent Bench

Personal agent benchmark results

These are the personal agent benchmark results from Personal Agent Bench v1: the share of errands each product completed in our first exam, with the 95% interval and the number of scored tasks beside every figure. In October, on the same 20 tasks, Instinct completed 40% (95% interval 20–64%), Muse 58% (36–77%) and Town 15% (5–36%); Pine has no October figure. These rates measure our replicas of each product with a substituted model, not the apps you download. There are two measurements, one from September and a newer one from October, all of them v1, and they are printed side by side, never merged, with no ranks and no combined score. The chart gives the picture; the tables below it hold every figure, the caveats and the data files.

Share of errands completed

11 October: all four given the same 20 tasks, once each. Not comparable with September.

  1. Muse58% (36–77)
  2. Instinct40% (20–64)
  3. Town15% (5–36)
  4. PineNo data, exam fault

The 09-21 to 09-22 records, filtered, 2026-09-22. All four answered the same tasks.

  1. Instinct52% (45–61)
  2. Muse40% (33–46)
  3. Town37% (30–44)
  4. Pine30% (23–36)

As of 2026-09-22. Each product answered its own draw, so rates are not directly comparable.

  1. Instinct43% (36–50)
  2. Pine20% (14–25)
  3. Muse19% (13–25)
  4. TownFigure withheld
  5. GrokExam pending

95% interval measured rateWhere two bands overlap, the gap is within measurement noise. These are product runtimes rebuilt for the exam with one substituted model, not the consumer apps measured end to end. Every figure, with its notes, below

What these personal agent benchmark results measure

Both months measure our replicas of each product, not the apps you can download. In September, the comparable 114-task batch ran a replica of each product's runtime with the exam's GLM 5.3 flash model swapped in, and the first batch used the exam's model route as well. Each product's interface reaches the exam through an adapter we wrote, and we have not verified that every production feature is reproduced exactly. These rates do not measure the consumer products end to end. The October batch works the same way: the same four replicas, rebuilt since September, on the same substituted model. Because the replicas are ours, a vendor changing its own app does not move these numbers; a change between months comes from our replicas, the tasks, the scoring rules or the exam. See model routing and its limits.

The v1 limitation, said plainly

Everything in the tables above is v1: the replicas of each product's runtime that we built for the exam did not reproduce every capability of the products themselves, so a rate here describes what our replica did on our substituted model, not what the consumer app would do on yours. v2 is a different exam on the same bench that answers that: its tasks run on real websites from network-level recordings, replayed so no request leaves the machine. One task is published in the open repository and there are no v2 results yet; how v2 works has the method.

Newest

v1 · October 2026

11 October batch, 20 tasks, plus 5 more batches listed separately. Every figure checked against the attempt records.

The October measurement. Each product answered the same 20 tasks once, under the exam rules that took effect at the end of September, and every attempt behind these rates has a record we checked the figures against.

ProductTasks completed95% intervalAttempts kept
Muse58%36 to 77%11 completed of 19 kept, over 20 tasks
Instinct40%20 to 64%6 completed of 15 kept, over 20 tasks
Town15%5 to 36%3 completed of 20 kept, over 20 tasks
PineNo data: almost none of its submissions reached the exam, so the column was set aside rather than scored as zero.
Not comparable with the September tables: the tasks, the scoring rules, our replicas of the products and the exam itself all changed in between, so a different rate is not a product getting better or worse. Each attempt ran once, so one attempt more or less moves a rate by five to seven points. Pine has no figure because its submissions did not reach the exam in this batch; the cause is not yet known and may be on the exam side.
Attempts were kept only when the exam did not break, the task suited the product, the product’s submissions were reaching the exam at all, and the attempt actually entered the exam. Each exclusion is counted in the records.
The other 5 October batches

The exam team ran five more batches in October, each with its own tasks, exam code and product builds. They are listed separately and never added together. Several were hit by exam faults found and fixed during the week (a sandbox wrongly judged broken, an expired certificate at the exam entrance), and the fixes have not been rerun yet.

BatchInstinctMuseTownPine
14 October, second run7 tasks67%21 to 94% · 2 of 333%10 to 70% · 2 of 657%25 to 84% · 4 of 7No data: almost none of its submissions reached the exam, so the column was set aside rather than scored as zero.
14 October, first run7 tasks0%0 to 66% · 0 of 233%10 to 70% · 2 of 6No data: almost none of its submissions reached the exam, so the column was set aside rather than scored as zero.No data: almost none of its submissions reached the exam, so the column was set aside rather than scored as zero.
12 October, large batch72 tasks19%9 to 38% · 5 of 2621%11 to 36% · 8 of 39No data: almost none of its submissions reached the exam, so the column was set aside rather than scored as zero.No data: almost none of its submissions reached the exam, so the column was set aside rather than scored as zero.
11 October, rerun20 tasks41%22 to 64% · 7 of 1733%14 to 61% · 4 of 1226%12 to 49% · 5 of 19No data: almost none of its submissions reached the exam, so the column was set aside rather than scored as zero.
11 October, pilot20 tasks50%30 to 70% · 10 of 2037%19 to 59% · 7 of 19No data: almost none of its submissions reached the exam, so the column was set aside rather than scored as zero.25%11 to 47% · 5 of 20

v1 · September 2026

Two measurements from 21 and 22 September: the first batch, and the same 114 tasks for all four. Supplied as totals, without attempt records. Open to read.

First batch, 2026-09-22

ProductTasks completed95% intervalScored tasksRead it how
Pine20%14–25%167More tasks scored than any other product. The September report said some tasks were being re-run; October status is unverified.
Instinct43%36–50%159159 scored tasks; Pine has the largest published sample at 167. Marginal intervals alone do not establish a significant lead.
Muse19%13–25%90This is the figure after we fixed our own adapter. The September report said a further 83 tasks were being re-run; October status is unverified.
Town———Withheld. The figure we were given does not reconcile with the only records we can check: five batches for this product survive in the bench repository's history and all five were refused for incomplete domain coverage, giving roughly 13% across 30 scorable tasks. We are not publishing a number while our own evidence disagrees with it. Asked and pending.
Grok———In the exam, no published figure yet. We have not taken this one apart at the runtime level either, so it has scores coming but no teardown.
As of 2026-09-22Source ZentorSorted by sample size, not by rate
Every figure is a share of tasks completed, with a 95% interval and the number of scored tasks it rests on. There is no combined score and no ranking, because a rate without its interval and its sample size is not a result. These September rates describe adapted runtimes using the exam model, on differing scored subsets; they are not consumer-product end-to-end performance.
These figures come from the team running the exam. The bench repository deliberately stopped tracking generated batch records in September, on the grounds that it holds the inputs a run needs rather than the outputs a run produces, so there is no published record behind these numbers for us or anyone else to check them against. We are printing them anyway, with that said plainly, because the alternative is printing nothing and pretending we have no numbers. If the underlying batches are ever published we will reconcile against them and say what changed.

The 09-21 to 09-22 records, filtered, 2026-09-22

A second measurement, kept beside the first rather than replacing it. Every product answered the same 114 tasks, which the table above does not: those rates rest on different tasks as well as on different sample sizes, so they were never strictly comparable with each other. These are replicas of product runtimes using the same substituted GLM 5.3 flash model, not shipping consumer products measured end to end.

ProductTasks completed95% intervalAttempts kept
Instinct52%45 to 61%380 over 114 tasks
Muse40%33 to 46%637 over 114 tasks
Town37%30 to 44%513 over 114 tasks
Pine30%23 to 36%546 over 114 tasks
Not a fresh result. These are older builds judged under older rules, filtered down from roughly 5,600 attempts by eight rules, each with its own count. The filter that drops attempts which never reached the exam removes over 400 from Pine and over 400 from Muse, and if those were the products declining to act rather than a broken entry, Pine lands near 22% and Muse near 24%.
Supplied transcripts have not been republished. The delivery contained 1649 transcript files for 2076 attempts, with 712 attempts sharing files after filename collisions; complete per-attempt provenance is not established.
Both months in one file: download CSV · JSON · also on Hugging Face (CC BY 4.0)
To cite these figures: Personal Agent Bench (Zentor), “Personal agent exam results”, as of 2026-10-11, https://personalagentbench.com/results/. The CSV and JSON above hold the same figures, one row per product per batch.

A completion rate on its own is not a result; the interval beside it says how much of that number is measurement and how much is luck, and the task count says how much weight the interval can carry.

The errands behind these rates run in stateful worlds that can be replayed, and judging reads the final state rather than what the product said it did. How the exam works sets out the worlds, the five ways a run can end, and why the measurements are published side by side instead of added up. The September rates were measured on 22 September; the exam's rules changed a week later, and what changed is on the record. The October batch was run under the new rules.

How the October personal agent results differ

The October figures are the first we can check attempt by attempt. The exam team handed over the record of every attempt, and we recomputed each rate, interval and exclusion count from those records; all of them matched. The intervals are Wilson score intervals, which stay sensible on small samples like these. The September figures never had that: they came to us as totals. The records themselves, grouped by kind of errand with each conversation and grader's note, are on the recorded AI agent examples. What else changed in October, from product launches to our own corrections, is gathered in the October 2026 personal agent report.

The headline October table is a single batch from 11 October, in which all four products were given the same 20 tasks. Five other October batches are listed under it and never added in, because each used different tasks, exam code and product builds, and the exam team asks that they not be pooled. We picked this batch because it is the largest one in which most of the products have figures and the fewest attempts were thrown out. It is still a trial run, and its intervals are wide.

Pine has no October figure. In every batch after the first pilot, its submissions in the web pages never reached the exam, and the cause has not been found; it may be on the exam side. That is a gap in the measurement, not a score of zero.

Two personal agent results that need saying out loud

Town has no figure in the first batch; the separate 114-task batch includes it. The supplied first-batch figure disagreed with five surviving batches, all refused for incomplete domain coverage, at roughly 13% across 30 scorable tasks. The supplied figure was more than three times that on 21 tasks. We withheld it pending reconciliation.

Muse's earlier figure was our fault, not the product's. An earlier batch put it at 12%. Then we found four defects in our own adapter. It knew two of the three words a confirmation arrives as, it merged pending interactions from two sources and could answer one twice, it kept a duplicate answer branch on the event stream, and it had no branch for a handover request, so a product correctly refusing to act unattended was recorded as a failure. The 19% is the figure after those were fixed. We are publishing the broken number as well as the fixed one, because a bench that quietly replaces its own bad measurements is asking you to trust something you cannot check.

A personal agent in the exam without a figure

Grok is in the exam with no published figure yet, and we have not taken it apart at the runtime level, so it appears in the results and not in the teardowns. What is public about it is on the Grok Bot page.

Reading personal agent results next to the teardowns

The results say how often something finished the job. The teardowns say what it can reach, what it keeps, and what you can check afterwards. A product can score well and still be one you would not hand your accounts to, and nothing in a completion rate will tell you that. The scorecard puts both on one page, and how we score explains why they are never folded into one number.

FAQ

Which personal agent did best in the benchmark? We do not rank them. In the October batch three products have figures. Instinct's interval (20 to 64%) overlaps both of the others. Muse's (36 to 77%) and Town's (5 to 36%) only just fail to meet, which on 19 and 20 attempts in a single trial batch is thin ground for an order. Where two intervals overlap, the gap between the rates is within measurement noise, and the first batch gave each product its own draw of tasks, so those rates are not directly comparable. The 114-task batch is the comparable one; read its intervals before reading anything into the order.

Why did Muse go from 19% in September to 58% in October? It is not the same measurement. The tasks, the scoring rules, our replica of Muse and the exam all changed in between, and the October figure rests on 19 attempts with an interval of 36 to 77%. Read it as a new measurement, not as Muse improving.

Are these the real consumer apps? No. They are replicas of each product's runtime driven through adapters, with the exam's model substituted. They tell you how the runtime arrangement copes with the errands, not how the app on your phone performs end to end.

Can I download the data? Yes. The CSV and JSON files linked under the first table hold the same figures, one row per product per batch, with the date they were measured. They are published under CC BY 4.0: reuse them freely, with a credit to Personal Agent Bench and a link to this page.