Personal Agent Bench

OSWorld benchmark: 108 long workflows, and where agents stop

The OSWorld benchmark gives an agent a real desktop, real applications and a task that a person would recognise as work, then checks the machine afterwards to see whether the work got done. Version 2.0, published in June 2026, rebuilt the task set around long workflows: 108 of them, each taking a skilled human a median of about 1.6 hours. It is the closest thing in this section to what we measure, and it is also the benchmark that has published the most uncomfortable finding about how far agents currently get.

  • On the OSWorld 2.0 board's default view, read on 5 November 2026, the top entry completes 44.33% of the 108 tasks outright. Most OSWorld scores you see quoted come instead from OSWorld-Verified, the 369-task original, whose top entry reached 86.0% in August 2026. The two can't be compared.
  • Its tasks are long. Nearly 70% take a skilled person more than an hour, and one shown on its own site runs past 500 agent steps without the claim ever being submitted.
  • It reports two numbers per entry, finished and partially finished, and they differ by more than thirty points at the top of the board.
  • Above a certain task length, every model it tested completes zero tasks outright. That is a published result, not an inference.
  • Its own scores move by double digits when only the benchmark version changes, which is the strongest available argument for printing dates beside numbers.

What the OSWorld benchmark actually measures

An agent is placed in a working operating system with real software installed and given a task described the way a person would describe it. When the run ends, scripts inspect the machine: is the file there, does it contain the right thing, was the form submitted. Success is a state of the world, not a judgement about the agent's explanation of itself.

Version 2.0 spans seven professional domains and 21 sub-categories, covering research, creative production, engineering, personal services, business and finance, administration and compliance, and healthcare. The authors also map tasks onto occupation families to estimate what share of actual paid work the set covers, which is an unusual thing for a benchmark to attempt and a useful corrective to task sets assembled from whatever was easy to automate. The largest shares land on document preparation, software and databases, and finance and operations analysis, with a long tail behind them. Reading that distribution is a quicker way to understand what the benchmark is for than reading the task list.

The jump from the first version is the headline. A version 1.0 task took roughly 30 tool calls. A version 2.0 task averages 318. That is not a harder version of the same exam, it is a different exam, and the authors treat it as one by keeping the first version online at its own address instead of replacing it.

OSWorld 1.0 and OSWorld-Verified: the version most scores refer to

Most OSWorld numbers you will see quoted are not from 2.0. They come from the original OSWorld, released in April 2024 by researchers at the University of Hong Kong, Salesforce Research, Carnegie Mellon and Waterloo: 369 tasks on real desktop and web apps, mostly on Ubuntu, each with its own checking script. Its paper put people at 72.36% and the best model at 12.24%.

In July 2025 the authors re-released it as OSWorld-Verified, fixing tasks the community had reported as broken and re-running the results. That board has since run past the human figure. In the results file behind it, read on 5 November 2026, the top entry is Anthropic's claude-fable-5 at 86.0%, dated 1 August 2026, with claude-opus-5 at 83.4% and Meta's Muse Spark 1.1 at 80.7% behind it, all at a 100-step budget. When we last read it on 12 October the top had been Intelligence-Indeed Agent at 90.19%; that entry is no longer on the board.

So when one source says an agent scores about 80% on OSWorld and another says 44%, both can be right. The first is OSWorld-Verified, a benchmark that has reached its ceiling much as GAIA has. The second is OSWorld 2.0, a different and much longer exam. The two cannot be compared, and a score quoted without its version tells you very little.

The OSWorld 2.0 leaderboard

The table follows the board's default view: the Foundation E2E GUI tab, which holds models driving the desktop directly rather than agent frameworks built on top of them, on the full 108-task set at a 500-step budget, across all task releases. In that view the second row is Meta's MuseSpark 1.3 at 39.25%; the trajectory file it links is labelled 106 tasks, so that row may not cover the full 108. Muse Spark is the model Meta launched its Muse agent on, and Muse is a product we took apart; a model's score here says nothing directly about how that product handles your errands. Switching to the board's All tab adds agent frameworks: Simular's Sai, running on Claude Opus 5, reaches 41.67% at 500 steps. Salesforce's Darwinx, on Claude Opus 4.8, reaches 44.44%, but at a 450-step budget, so it shows only once the step-budget filter is changed from 500. The Offline set, a subset of tasks that need no internet, gives higher numbers again.

#SubmissionBenchmark releaseOverallPartial
1Claude Opus 5 (Max, batch tool)v2.144.33%77.67%
2MuseSpark 1.3 (Max, batch tool), Metav2.139.25%69.56%
3Claude Opus 5 (High, batch tool)v2.136.89%68.92%
4Claude Opus 5 (Xhigh, batch tool)v2.133.33%73.11%
5Claude Opus 5 (Medium, batch tool)v2.133.01%64.85%
6Claude Opus 5 (Max, batch tool)v2026.08.0831.43%68.31%
Binary accuracy on OSWorld 2.0, the share of the 108 long-horizon workflows completed outright, with the partial column giving the share of intermediate checkpoints reached. Read on 2026-10-22 from the results file behind the board (official-results.json, updatedAt 2026-10-15), filtered the way the board's default view is: the Foundation E2E GUI tab (models driving the desktop directly, no agent framework on top), the full 108-task set, a 500-step budget and all task releases. Other views give other numbers. Under the All tab, which adds agent frameworks built on these models, the leader at 500 steps is still Claude Opus 5 (Max) at 44.33%, and Salesforce's Darwinx with Claude Opus 4.8 scores 44.44% at a 450-step budget, visible only with the step-budget filter off 500. The Offline set, a subset of tasks that need no internet, puts Claude Opus 5 (Max) at 48.65%. The board carries no confidence intervals; its cost column is filled for some entries (Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.5, Qwen 3.7-Plus, MiniMax M3, Kimi 2.6, and Simular's Sai under All) and empty for the leading entries. MuseSpark 1.3's linked trajectory file is labelled 106 tasks. The release column is kept because it moves the score: the same model at the same setting scores 44.33% on v2.1 and 31.43% on v2026.08.08.

In OSWorld, finished and nearly finished are not the same column

Every entry carries two rates. Binary accuracy is the share of tasks completed outright. Partial is the share of checkpoints reached.

At the top of the default view those two numbers are 44.33% and 77.67%. The gap is the whole story. About 78% of scored intermediate checkpoints are reached, and fewer than half of the tasks actually land.

That gap is not a rounding problem, it is the shape of the failure. An agent that gets most of the way through a reimbursement claim and never submits it has not done 77% of the errand. It has done none of it, and it has probably left a half-filled form behind that someone now has to find and clean up. A benchmark that reported only the partial figure would describe these agents as nearly competent. A benchmark that reported only the binary figure would hide how close they get. Printing both is the honest option, and it is the one chosen here.

The point where OSWorld completion goes to zero

The site publishes a breakdown of completion against task length, and it ends somewhere most benchmark pages do not go.

On tasks under 45 minutes of human-equivalent work, the better models complete 20% to 24% outright. In the 137 to 163 minute band, no model exceeds 10%. Above 163 minutes, binary completion falls to zero for every model retained in the plot.

Not low. Zero. In the reported cohort and settings, no plotted model completed a task in the longest duration band.

The authors attribute it to compounding: execution load and state-management errors accumulate over a long run in a way that short tasks never expose. The dangerous failure is rarely a single wrong click. It is an early misreading that survives twenty steps and quietly invalidates everything after it.

What a 500-step failure actually looks like

The site walks through one run in detail, and it is worth reading because it is the most legible account of long-horizon failure published anywhere in this section.

The task is an overseas reimbursement claim: gather evidence from a bank statement, an email and a guidelines document, then file it in the expense system before a deadline. The agent reads the rules correctly at step 65. By step 120 it is still searching for evidence. At step 220 it hits a dead API and turns that into a tooling detour. At step 300 it is still collecting. It finally opens the expense system at step 459, and the run ends with the agent reverse-engineering a date picker rather than with a submitted claim.

The trace shows planning, tool-use and state-management problems; it does not establish that every individual action was competent. What is missing is any sense that the preparation has a budget, and that is not a skill a resolution rate measures.

The same shape is easy to picture in an errand. An assistant asked to cancel a subscription reads the terms, finds the help centre, locates the policy, explains the policy back, and never reaches the button. Our exam scores whether the cancellation actually happened, not how well the policy was explained. Thoroughness and completion are separate abilities, and only one of them is what was asked for.

OSWorld version numbers move the score as much as reasoning effort does

One pair of rows on that board deserves more attention than the top one.

The same model, at the same reasoning setting, with the same tool configuration, scores 44.33% on one release of the benchmark and 31.43% on another. Nothing about the agent changed. Almost thirteen points of difference come from which version of the task set it was scored against.

This is the strongest available evidence for a rule we apply to our own numbers: a rate without a date and a method beside it is not a measurement, it is a rumour. Anyone quoting an OSWorld score without saying which release produced it is quoting a number that could be off by a third. The board itself handles this correctly, letting you filter by release version, which is more than most leaderboards offer and more than most readers use.

Buying the last few points costs more than it looks

The published findings include a curve that is easy to summarise and hard to argue with: scores climb, and the token budget required to climb them climbs faster.

One model reaches about 14% binary completion on roughly 39 thousand output tokens and then flattens out. Another gets to 18.2% at 150 thousand, and a later release of it reaches 20.6% at 244 thousand. The latter point uses more than six times the output tokens of the first, across different model configurations.

This matters to anyone building on top of these models rather than reading about them, and it matters in a specific way for a personal agent. An errand is not worth an unbounded budget. The plotted points compare different models and releases. They do not measure flight-rebooking probabilities, isolate the effect of extra tokens, or establish how consumer products choose their budgets. The shape of that curve is an argument for measuring at a fixed budget, which is what both this benchmark and our own exam do.

Where OSWorld stops being about personal agents

OSWorld is the closest neighbour this site has among the agent benchmarks, and the distance is still real.

Its tasks are professional work: engineering, compliance, finance, healthcare administration. They are long and they are hard, but they are assigned. Version 2.0 includes hidden state and seeded user-history context. That is different from measuring how an agent learns a continuing user's preferences across errands or handles consequences in that user's real accounts.

The machine is also fresh at the start of every task. The products on this bench carry state between errands on purpose, and the interesting failures are the ones where yesterday's context produces today's wrong answer. Our scorecard calls that axis where memory lives, and OSWorld has nothing to say about it by design.

Three things from it carry over to how we measure.

Report completion and progress separately rather than blending them. Keep the horizon in the measurement, because an errand that is abandoned at step 459 is a different failure from one that is refused at step 2, and a single rate erases the difference. Put the version and date of the task set next to every number, since this benchmark has now demonstrated what happens when you do not.

What we do not take is the step budget as a ceiling. A personal agent that runs out of patience mid-errand does not stop cleanly, it leaves a half-booked flight behind, and that consequence is something our scoring has to catch rather than treat as a timeout.

For another examination of the original tasks, Epoch AI’s OSWorld analysis discusses ambiguous instructions and changes over time. OSWorld-Human studies efficiency as well as completion, a separate question from who leads this table.

FAQ

How many tasks does OSWorld 2.0 have? 108 long-horizon workflows, spanning seven professional domains and 21 sub-categories.

How long are the tasks? The median takes a skilled human about 1.6 hours, and 69.6% take more than an hour. An average run uses around 318 tool calls, against roughly 30 in version 1.0.

What is the difference between binary accuracy and partial? Binary accuracy is the share of tasks completed outright. Partial is the share of intermediate checkpoints reached. At the top of the default view they are 44.33% and 77.67%.

Is the older version still relevant? Yes, because most quoted OSWorld scores come from it. The original 369-task set was re-released as OSWorld-Verified in July 2025, and its top entry reached 86.0% in August 2026. Scores from it are not comparable to 2.0 scores, and the two live on separate sites.

What is the human baseline on OSWorld? The original paper reports people completing 72.36% of the 369 tasks. OSWorld 2.0 measures human effort as time instead: its median task takes a skilled person about 1.6 hours.

Does it publish confidence intervals? No. The board reports point estimates with no margin, so small differences between adjacent rows cannot be distinguished from run-to-run variation. The cost per task column is filled for some entries, including Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.5 and several open models, and for Simular's Sai under the All tab, but it is empty for the leading model entries. So the board cannot yet be used to compare price across the top of the table.

Is a desktop benchmark relevant to a personal assistant? Partly. Several products on this bench do their work on a machine of their own, so the skill being measured is real. Seeded history and task instructions do not establish continuing-user learning across errands, and the benchmark accounts do not measure effects on a real user's accounts.

The other benchmark pages

No numbers invented. Personal agents taken apart at the runtime level, on a method built to take more.

See the scorecard