---
title: "AssistantBench: web tasks, scoring and leaderboard"
description: "AssistantBench: 214 web tasks that let an agent decline to answer, its leaderboard, how it differs from Assistant Benchmark, and what it misses."
url: https://personalagentbench.com/benchmarks/assistantbench/
last_updated: 2026-10-16
---

# AssistantBench: web tasks, scoring and leaderboard

AssistantBench is an academic benchmark for web agents, [posted in July 2024](https://arxiv.org/abs/2407.15711), [published at EMNLP 2024](https://aclanthology.org/2024.emnlp-main.505/) and built at Tel Aviv University with collaborators at Penn, the Allen Institute, Washington and Princeton. It holds 214 tasks that require browsing more than 525 pages across 258 websites, and it asks a single question: can an agent go out onto the open web and come back with the right answer. It is on this list partly because its name is one word-break away from Assistant Benchmark, a consumer scorecard we cover on [the benchmarks page](https://personalagentbench.com/benchmarks/), and the two get confused. The better reason is that it does one thing with its scoring that we think is correct and that most web benchmarks do not do.

- It is a question-answering benchmark on the live web. The agent finds things. It does not change anything.
- It lets an agent decline to answer, and reports that separately, which rewards admitting failure instead of inventing a result.
- Its headline accuracy carries partial credit, and the exact-match figure beneath it is roughly half as large.
- Its official leaderboard mixes finished systems with leftover test submissions, and on 5 November 2026 its hosted page was still down with a runtime error. Princeton's HAL runs its own AssistantBench board on the 33 public development tasks, led by o3 at 38.81%.
- It is not Assistant Benchmark (assistantbenchmark.com), which scores consumer assistants by hand. Different project, different question.

## What AssistantBench actually measures

A task is a realistic information-gathering request that would take a person a while: the kind of thing where the answer exists on the web but is spread over several pages and needs assembling. The agent browses, gathers, and produces an answer. Scoring compares that answer against a gold answer, with credit available for being partly right on structured answers rather than only for an exact string.

The tasks came from people rather than templates: the paper says 53 people, 35 of them domain experts, shared recent tasks that were time-consuming and needed expertise, and those became the 214. The displayed leading exact-match rate on the official board is 27.1%, so nothing about it looks saturated. The split matters for reading the board: [the public dataset](https://huggingface.co/datasets/AssistantBench/AssistantBench) holds 33 development tasks with their answers and 181 test tasks with the answers left blank. The leaderboard scores the 181. The accompanying paper also introduces an agent of the authors' own, built by adding planning and memory components to an existing web agent, which tells you what they believed the hard part was.

## The AssistantBench leaderboard

| # | Submission | Organisation | Overall | Exact match |
| --- | --- | --- | --- | --- |
| 1 | Autobot ChatGPT-HQ + GPT-5.6 Sol Ultra | Autonomous Production | 50.70 | 27.1 |
| 2 | Jina + Claude Code | Om Labs | 42.10 | 22.7 |
| 3 | Distyl Button | Distyl AI Inc. | 38.00 | 19.9 |
| 4 | TongAgents | BIGAI | 37.60 | 18.2 |
| 5 | test_Agent | not stated | 36.70 | 18.2 |

As of **2026-10-08** · [AssistantBench leaderboard](https://huggingface.co/spaces/AssistantBench/leaderboard) · [AssistantBench paper](https://arxiv.org/abs/2407.15711)

Accuracy over the 181-task test split (the other 33 of the 214 tasks are a development set), which awards partial credit on structured answers, with the exact-match column beside it. Read on 2026-10-08. The hosted page did not render a table in an ordinary browser session, so the figures were taken from the application's own configuration, which is the data that page is built from. Re-checked on 2026-10-12 and 2026-10-16 through the Hugging Face API: both times the Space was in a runtime error, failing to load its results dataset, so these 8 October figures could not be refreshed. This is an open submission queue that has not been pruned: leftover test runs sit among finished systems, and three rows below these carry identical figures under different names. No confidence intervals and no dates are published.

## How AssistantBench lets an agent say it does not know

The full board carries three columns that do something unusual, though only accuracy and exact match made it into the table above. Accuracy is scored over all 181 test tasks. Answer rate is the share the agent chose to answer at all. Precision is accuracy among the ones it answered.

Splitting those apart changes the incentive completely. On a benchmark that scores only accuracy, a guess is free: a wrong answer costs the same as no answer, so the optimal strategy is to always produce something. Reporting answer rate separately makes abstention visible and makes a confidently wrong answer look worse than an honest refusal.

We can't give you the leader's answer rate and precision. We checked again through the Hugging Face API on 12 and 16 October and on 5 November 2026: the [leaderboard Space](https://huggingface.co/spaces/AssistantBench/leaderboard) reports a runtime error, and its own error log says the app fails at startup because it cannot load the results dataset it reads from, which the Hub reports it cannot find; that does not establish whether it is private or removed. We would be guessing, so we don't.

The paper's own baselines show the trade instead. In [its main results table](https://arxiv.org/html/2407.15711v2), run with GPT-4 Turbo on the test set, a model answering from memory with one worked example answered 89.5% of tasks with 24.8 precision, for 22.2 accuracy. The authors' web agent, SPA, answered only 35.9% but with 30.9 precision, for 11.1 accuracy. The older SeeAct agent answered 15.5%. The memory-only model looks twice as good on accuracy and is wrong more often when it does answer; the paper's own summary is that closed-book models "tend to hallucinate facts." That gap is the whole reason the extra columns exist.

That is the right design, and it is the single most transferable thing on this page. The equivalent failure for a personal agent is worse than a wrong answer to a question. An assistant that cannot find your booking reference and invents one has not merely failed, it has handed you something that looks like success and will not be discovered until it matters. Our [exam protocol](https://personalagentbench.com/how-the-exam-works/) records whether the agent reported its actions truthfully as a separate measurement from task success, for exactly this reason, though the published September aggregates predate that reply grading.

## Partial credit in AssistantBench, and what the exact-match column says

The leading entry scores 50.70 on accuracy and 27.1 on exact match.

Both are real numbers about the same runs. The first says the agent produced answers that were substantially right about half the time. The second says it produced the precisely correct answer about a quarter of the time. Which one you should care about depends entirely on what you were going to do with the answer, and a headline that quotes only the first is not wrong so much as incomplete.

The difficulty breakdown underneath makes the same point more sharply. That same entry scores 88.0 on easy tasks, 67.7 on medium and 39.7 on hard. An average across those three is a statement about the mix of the task set as much as about the agent.

## What the AssistantBench boards look like now

This part needs stating plainly, because a leaderboard position is the kind of thing that gets quoted.

The board is a public submission queue, and it has not been pruned. The entry sitting fifth is named test_Agent and lists no organisation. Below it are rows called reb_1, reb3, reb4, check_1, check_state, tt_api_1, wsg_v1 and several variants of SO_R3_Testing. Three separate rows carry identical accuracy, exact-match and difficulty-breakdown figures while differing only in answer rate, which may indicate duplicate submissions; we cannot establish whether the systems are identical.

None of that is misconduct. It is what an open submission form looks like after two years without a janitor, and the underlying scores are still produced by the benchmark's own scorer rather than self-reported. But a position on this board is a position among submissions, not among products, and the distance between those two things is large here.

The [hosted leaderboard](https://huggingface.co/spaces/AssistantBench/leaderboard) did not render a table in a normal browser session on 8 October 2026, and on 12 and 16 October and 5 November Hugging Face listed it in a runtime error. The figures above were read on 8 October out of the application's own configuration, which is the same data the page is built from. We record that because the alternative would be to quote numbers without saying how we got them.

With the official board down, the one we could read is the [AssistantBench page on Princeton's Holistic Agent Leaderboard](https://hal.cs.princeton.edu/assistantbench), read on 16 October 2026. It is a different measurement in three ways. It runs only the 33 development tasks whose answers are public, not the 181 test tasks. It runs every model through one scaffold, Browser-Use, and marks each row as reproduced by the HAL team. And it prints the cost of each run beside the accuracy.

Its best entry was o3 at medium effort, 38.81% for $15.15, ahead of GPT-5 at medium effort, 35.23% for $41.69. Claude Opus 4.1 at high effort scored 13.75% for $779.72. HAL says it has paused adding new models and that some reported costs are overstated by exactly twice, from a trace-counting bug it disclosed on 5 October 2026 and has not yet corrected, so treat the dollar figures as upper bounds. Even so, costs running from $2.18 to $779.72 across its 15 runs, with the most expensive run nowhere near the top, are the kind of thing the official leaderboard never showed.

## A small fixed task set ages in a particular way

A fixed set of 214 tasks on the live web requires checking whether its reference answers remain current.

The answers live on real websites, where pages, listings and prices can change. A fixed answer key can therefore diverge from current evidence. The displayed scores do not tell us whether any particular failure reflects agent error or a changed source; that requires task-level review.

The authors know this. In October 2024 they posted [a file that tags every one of the 214 tasks](https://huggingface.co/datasets/AssistantBench/time_dependency) by time dependence: 93 marked "static", 33 "stable" and 88 "unlikely". The file carries no explanation of the labels and the leaderboard does not use them, so it classifies possible time dependence without measuring actual answer drift or its effect on scores.

The same small size makes the top of the board volatile. A single difficulty band holds enough tasks that a handful of items decides several points of accuracy, and with no confidence intervals published, adjacent rows cannot be told apart. This is the same objection we make to our own early figures, which is why the rates on our scorecard carry a range and a count rather than standing alone.

That risk does not explain the leading entries by itself. The top two are commercial agent stacks rather than the paper's baselines, and they score well above the academic baselines the benchmark shipped with. A fixed public set can be exposed to contamination or task-specific tuning. The board alone does not establish whether either affected these scores or how well they generalise.

## Where AssistantBench stops being about personal agents

AssistantBench is a retrieval benchmark wearing an assistant's name.

Every task ends with an answer. Nothing is booked, cancelled, sent, paid for or undone. There is no account to sign into, no message that reaches another person, no action that cannot be taken back. The hardest thing on the board is finding something difficult on the open web and reporting it accurately, which is a genuine skill and a small fraction of what an assistant is asked for.

The tasks are also one-shot. Each arrives with no history behind it and leaves no trace after it. The products this site examines are supposed to carry a preference you stated last month into a decision made today, and the most interesting things they get wrong come from that carry-over rather than from any single lookup. Our [teardowns](https://personalagentbench.com/agents/) spend most of their length on it.

## AssistantBench, Assistant Benchmark and the newer assistant benchmarks

Names in this area are close, and that causes two kinds of confusion.

The first is with Assistant Benchmark, at assistantbenchmark.com, which is a different thing entirely: people testing consumer assistants from their own accounts and scoring each task by hand. We cover it, and why its board and another one disagree, on [the benchmarks page](https://personalagentbench.com/benchmarks/). AssistantBench, this page, is a fixed set of web questions for research agents. A 50% accuracy figure here says nothing about half your errands getting done.

The second is with a wave of newer research benchmarks that also put "assistant" at the centre, and that often come up for the same search:

- [π-Bench](https://arxiv.org/abs/2605.14678) (May 2026) tests proactive personal assistants: 100 multi-turn tasks across five user personas, with needs the user never states outright and continuity between sessions.
- [AstaBench](https://arxiv.org/abs/2510.21652) (October 2025), from the Allen Institute, is a suite for scientific research agents that controls for cost and tool access.
- [JobBench](https://arxiv.org/abs/2605.26329) (May 2026) covers 130 tasks across 35 occupations, chosen from what experts say they want to delegate; the best of 36 models it tested reached 45.9%.
- [LiveClawBench](https://arxiv.org/abs/2604.13072) (March 2026) runs assistant tasks against stateful mock applications, and [HiL-Bench](https://arxiv.org/abs/2604.09408) (April 2026) tests whether coding and SQL agents ask for help when a requirement is missing or ambiguous.

Of those, π-Bench is the closest to a personal agent, because it carries context across sessions. None of them acts on real accounts.

## What we took from AssistantBench and what we left

The abstention split is in our protocol. Task success is read from the state of the world, not from the agent's reply, so an assistant that reports it could not complete an errand is recorded as not having completed it, and whether an assistant's report of what it did was truthful is graded separately. Those are not the same failure, and averaging them together destroys the distinction a reader most needs. The published September aggregates were produced before reply grading was added, so they carry only the first half.

What we left is the fixed public task set. AssistantBench publishes every test question and hides only the answers behind its scorer. Publishing every task would let anyone reproduce our numbers, which is the argument for it, and would also let a vendor tune against them, which is the argument against. Our compromise is to publish the method and the [per-run records](https://personalagentbench.com/runs/), now for four products, while keeping the live task set closed, so those individual attempts can be checked; aggregate figures still lack public batch evidence. That is a weaker form of reproducibility than this benchmark offers when its scorer is up, and it is a deliberate trade rather than an oversight. The outage shows the cost of the other design too: with test answers private and the hosted scorer down, the public hosted scoring route is unavailable. Maintainer or offline access was not ruled out.

## FAQ

**How many tasks does AssistantBench have?** 214, drawn from more than 525 pages across 258 different websites. 33 are a development set with public answers; the other 181 are the test set the leaderboard scores, with answers withheld.

**What is the difference between accuracy and exact match?** Accuracy awards partial credit on structured answers. Exact match requires the precisely correct answer. At the top of the board those figures are 50.70 and 27.1 for the same runs.

**What does answer rate mean?** The share of tasks the agent chose to answer rather than abstain on. Precision is accuracy calculated only over the tasks it answered, so an agent that declines the ones it cannot do scores better on precision than on accuracy.

**Can anyone submit to the leaderboard?** Yes. Submissions are prediction files scored by the benchmark's own scorer. The queue has not been cleaned up, so finished systems and leftover test runs sit on the same board.

**Is it still maintained?** The leaderboard application was last updated on 13 September 2026, and on 12 and 16 October 2026 Hugging Face showed it in a runtime error because the app could not load its results data. The task dataset on Hugging Face was last modified in July 2024; the only later addition we found is the October 2024 time-dependence file.

**Is there another AssistantBench leaderboard?** Yes. Princeton's HAL runs one on the 33 development tasks with a single scaffold and prints cost per run. On 16 October 2026 it was led by o3 at 38.81%. Its numbers are not comparable with the official board's test-set figures.

**Is AssistantBench the same as Assistant Benchmark?** No. AssistantBench is a 2024 academic set of web questions. Assistant Benchmark (assistantbenchmark.com) is a separate site that scores consumer assistants by hand from real accounts.

**Does it test anything a personal assistant does?** It tests finding information on the live web, accurately, without inventing an answer. That is a real part of the job. It does not test acting, remembering, or anything irreversible.

## The other benchmark pages

- [**GAIA**Every question arrives cold. Nothing carries over from the last one, so a system that remembers you scores exactly the same as one that doesn't.](https://personalagentbench.com/benchmarks/gaia/)
- [**BrowseComp**Reading, not doing. Nothing is booked, cancelled, paid for or sent.](https://personalagentbench.com/benchmarks/browsecomp/)
- [**WebArena**The sites aren't yours and hold nothing about you. No saved cards, no order history, no account that remembers.](https://personalagentbench.com/benchmarks/webarena/)
- [**OSWorld**Skill, not memory. It asks whether an agent can operate the machine, never whether it knows why you'd want it to.](https://personalagentbench.com/benchmarks/osworld/)
- [**Tau-bench**The user is simulated and the domains are narrow, but this is the closest thing on the list to what a personal agent actually does.](https://personalagentbench.com/benchmarks/tau-bench/)
- [**AgentBench**None of the environments is your life. No calendar, no inbox, no bill to dispute.](https://personalagentbench.com/benchmarks/agentbench/)
- [**Terminal-Bench**Almost nothing a personal agent does happens at a terminal.](https://personalagentbench.com/benchmarks/terminal-bench/)
- [**SWE-bench**A closed problem with a written specification. An errand has neither, and it cannot be rolled back.](https://personalagentbench.com/benchmarks/swe-bench/)

No numbers invented. Personal agents taken apart at the runtime level, on a method built to take more.

[See the scorecard](https://personalagentbench.com/)

---

Published by Personal Agent Bench (https://personalagentbench.com), run by the Zentor team. Zentor builds agent software of its own, so its products are not on the scorecard and will not be added.
