Terminal-Bench 4.0: leaderboard, tasks and error bars
Terminal-Bench is a benchmark that puts an AI agent in a terminal and asks it to finish real work there: migrate the database, fix the pipeline, leave the machine in the state the task describes. It launched in May 2025, is hosted by Stanford, Harbor and the Laude Institute, and its public leaderboard scores version 4.0, a set of 66 tasks. Every row on that board carries a 95% confidence interval, which is rare enough in this field to be the main reason it has a page here, since it measures almost nothing a personal agent is for.
- Each task is a real job with a checkable end state, and some come with an eight-hour time limit. The harness inspects the machine afterwards.
- On 5 November 2026 the leader resolves 64.8% ± 3.1%. The gap to second place is smaller than the two margins put together, which is the argument for printing them.
- The board also prints cost. Two entries tie at 58.2%, one for about $3,270 and the other for about $630, each over 330 trials.
- There have been five scored versions in about fifteen months, from 1.0 to 4.0. A score without its version cannot be compared with anything.
- It measures work in a disposable machine. No inbox, no saved logins, no consequences outside the box.
What Terminal-Bench measures, with real 4.0 tasks
An agent gets a terminal inside a container and a task written the way an engineer would hand it over. It works, and the harness then checks whether the described end state was reached. Resolution rate is the share of attempts that got there.
The tasks are not puzzles. Two from the 4.0 set in the repository, read on 5 November 2026, show the scale. live-database-cutover asks the agent to move a running e-commerce API from MySQL to PostgreSQL with zero failed customer requests and no stale reads while traffic continues, keeping p95 latency within 10 milliseconds of the MySQL baseline. payments-pipeline-fix asks it to speed up worker startup in a transaction pipeline so overdraft notifications are neither missed nor duplicated and arrive within five seconds during rolling deploys. Both give the agent 28,800 seconds, eight hours. Other task names in the set include uefi-bootkit, formal-crypto and freecad-impeller, which gives a sense of the range.
The scored set is versioned, which matters more than it sounds. A benchmark whose task list quietly grows is a benchmark whose scores cannot be compared across months. The leaderboard definition in the repository names the board Terminal-Bench 4.0 and ties it to dataset version 4, and the task manifest on the main branch lists the same 66 tasks.
The Terminal-Bench 4.0 leaderboard
| # | Submission | Agent | Overall | Cost | Dated |
|---|---|---|---|---|---|
| 1 | Opus 5.5 (max) | Claude Code | 64.8% ± 3.1% | $4.7k | 2026-09-22 |
| 2 | Sonnet 5.5 (max) | Claude Code | 61.8% ± 2.9% | $7.3k | 2026-09-28 |
| 3 | GPT-6 Astra (max) | Codex | 58.2% ± 2.8% | $3.3k | 2026-09-03 |
| 4 | GPT-6.1 Sol (max) | Codex | 58.2% ± 3.1% | $0.6k | 2026-09-29 |
| 5 | Fable 5.1 (max) | Claude Code | 57.9% ± 3.8% | $6.2k | 2026-09-01 |
Read the Terminal-Bench interval before the rank
Read the top of that table as a ranking and you get a clean story: one model wins, another is second, two tie for third.
Read the intervals and the story loosens. First place is 64.8% with a margin of 3.1 points, so roughly 61.7% to 67.9%. Second is 61.8% with 2.9, so roughly 58.9% to 64.7%. The ranges overlap. Overlapping margins are not a formal test that two entries are equal, and a proper comparison would need the per-task outcomes and an explicit method, but they are a clear warning against reading a three-point gap as settled.
Fifth place shows the same thing from the other side. The table here prints one row. When we first read the board on 16 October it had four entries at 57.9%, each with 191 successes out of 330 trials: two configurations of Fable 5.1 and two of GPT-6 Astra. On 5 November a single Fable 5.1 row holds that spot.
The data behind the board holds one more number worth knowing. Alongside the per-trial rate it records how many tasks an entry solved at least once in five tries. Opus 5.5 at maximum effort and Fable 5.1 at maximum effort both solve 78.8% of tasks that way, though their per-trial rates are 64.8% and 57.9%. The difference between them is less about what they can do than about how often they do it, which is the question tau bench is built around.
Our own scorecard prints a range and a task count next to every rate and refuses to combine them into one number. We arrived at that rule separately, and finding it already in use here was the main reason to write this page.
What a Terminal-Bench run costs, and why that is on the board
Two columns on the official board are unusual: tokens and cost. Each row in the board's data is 330 trials on the 66 tasks, and the cost is the total for all of them. For the leader that is $4,718.65, or about $14 a trial, and the average trial took about 80 minutes.
Publishing that beside the result is an honest move. It makes visible that a score is partly a statement about budget.
The clearest example is not at the top. Two entries share third place at 58.2%. One, GPT-6 Astra, spent about $3,270 getting there; the other, GPT-6.1 Sol, about $630. Same resolution rate, same tasks, roughly a fifth of the money. Most benchmarks in this section publish a score and stop, so you would never see that. The cheaper entry also has a slightly wider interval, but nothing on the board shows that spending less caused it; the two differ in model and configuration as well as price.
Whether cost matters depends on who is reading. A lab comparing capability can ignore it. Anyone deciding what to run cannot, since the choice is between two bills, not two numbers. Once cost is on the page, first place stops being a claim about which model is best and becomes a claim about which is best at some price. Our own board has no cost column, because the four products on our scorecard are not billed in a way that compares.
Terminal-Bench versions: 1.0, 2.0, 2.1, 3.0 and 4.0
The project's own posts give the history:
- 1.0, May 2025: the first release, scored on a set called T-Bench-Core-v0.
- 2.0, 7 November 2025: a harder, re-verified set of 89 tasks, released with Harbor, the new harness. The Terminal-Bench 2.0 paper followed in January 2026.
- 2.1, 6 May 2026: fixes to 28 of the 89 tasks. Most agents scored higher on the fixed set, one by 12 points, which shows how much the broken tasks had been costing.
- 3.0, 30 July 2026: a new set of 74 tasks across seven domains, with the best models near 34% at release. It was a community effort, and Turing, one of the contributors, wrote about its share of the tasks on 21 September. That is the same 3.0, not a rival version.
- 4.0, 28 August 2026: a revision of 3.0 that recalibrated time and machine resources, fixed tasks and removed eight, leaving the 66 the current board scores.
So 2.0 numbers you find elsewhere are not comparable with these. A score against 2.0 and a score against 4.0 are answers to different exams. We have the same problem on our own board, where one product's rate moved seven points after we fixed defects in our adapter rather than because the product changed, and the scorecard says so in the row.
There is also Terminal-Bench-Science, a separate benchmark of research workflows with tasks written and reviewed by domain experts. Its 0.1 board lists 70 tasks. Its scores use a different task set and do not belong in the table above.
What the dates on each Terminal-Bench row are doing there
Every row carries the date its run was scored, and the five entries above span four weeks of September 2026. That column does more work than it looks.
A leaderboard without dates implies all its rows were produced under the same conditions, which is almost never true. Models get updated behind a stable name, the harness gets patched, provider latency and rate limits move. A row scored on 1 September and a row scored on 29 September are two measurements of a moving system, and the gap between them is not necessarily a difference between the models.
This is the most common way benchmark tables mislead without stating anything false. We hit it ourselves: one of our figures was produced before we fixed our adapter, and the row keeps the earlier figure beside the current one rather than quietly refreshing the number.
Where Terminal-Bench stops being about personal agents
Everything above is about a terminal. The task arrives cold, the agent has no history with you, and the machine was created for the task and is destroyed after it.
A personal agent fails in ways that setup cannot reach. It sends the message to the wrong thread. It accepts a retention offer on your behalf. It books a second reservation because the first page was slow. None of those are terminal problems, and none show up as a resolution rate.
There is one overlap worth naming. Several products on this bench give the agent a machine of its own, and in Muse's case a Linux VM per user. Everything Terminal-Bench measures about driving a shell applies there. The difference is that that machine persists, holds your files and carries yesterday into today, so a mistake does not vanish when the task ends. Terminal-Bench tells you how well a model drives a shell. It cannot tell you what happens when the shell is yours. That is the subject of our teardowns and of how the exam works.
How to run Terminal-Bench and check a score
The harness is open and the tasks are readable, which is less common than it should be. Anybody can open a task's instructions and tests and judge whether the scored behaviour is the behaviour they care about. A task you can open is a claim you can check.
For a reproducible comparison, start from the official repository and its submission guide, pin the dataset version and harness version, record model settings and effort level, and keep per-task outcomes and costs for every trial. Compare the same task IDs before reading anything into an aggregate difference. Expect it to be expensive: the leader's 330 trials cost about $4,700 in model fees. We did not run a new evaluation for this page.
FAQ
What is Terminal-Bench? A benchmark that gives an AI agent a terminal and a task with a checkable end state, then measures how often the machine ends up in the described state. It is hosted by Stanford, Harbor and the Laude Institute, and the current public leaderboard is version 4.0.
What is the top Terminal-Bench 4.0 score? On 5 November 2026 the leader is Opus 5.5 in Claude Code at maximum effort, resolving 64.8% ± 3.1% of 330 trials. Second is Sonnet 5.5 at 61.8% ± 2.9%, so the two margins overlap.
How many tasks does Terminal-Bench contain? 66 for version 4.0, which the current leaderboard scores. Version 2.0 had 89 and 3.0 had 74. The count comes from the dataset's page on the Harbor hub and matches the task manifest in the repository on 16 October 2026, and the board data on 5 November.
What is the difference between Terminal-Bench 2.0 and 4.0? They are different task sets. 2.0, from November 2025, had 89 tasks; 4.0, from August 2026, is a revision of the 74-task 3.0 set with eight tasks removed. Scores on one cannot be compared with scores on the other.
Is Turing's Terminal-Bench 3.0 a different benchmark? No. Turing contributed tasks to Terminal-Bench 3.0, which the Terminal-Bench team released in July 2026. Version 4.0 is a revision of that same set.
Does a good Terminal-Bench score mean a good personal agent? No. It measures operating a shell on a disposable machine. A personal agent is judged on what it does with your accounts, your messages and your money, and those failures have no terminal equivalent.
The other benchmark pages
- GAIAEvery question arrives cold. Nothing carries over from the last one, so a system that remembers you scores exactly the same as one that doesn't.
- BrowseCompReading, not doing. Nothing is booked, cancelled, paid for or sent.
- WebArenaThe sites aren't yours and hold nothing about you. No saved cards, no order history, no account that remembers.
- OSWorldSkill, not memory. It asks whether an agent can operate the machine, never whether it knows why you'd want it to.
- Tau-benchThe user is simulated and the domains are narrow, but this is the closest thing on the list to what a personal agent actually does.
- AgentBenchNone of the environments is your life. No calendar, no inbox, no bill to dispute.
- SWE-benchA closed problem with a written specification. An errand has neither, and it cannot be rolled back.
- AssistantBenchStill one-shot. No standing preferences, no routine that fires next Monday.
No numbers invented. Personal agents taken apart at the runtime level, on a method built to take more.
See the scorecard