---
title: "SWE-bench: Verified leaderboard, Pro and what it measures"
description: "SWE-bench explained: who built it, the 500-task Verified leaderboard on 5 November 2026, SWE-bench Pro scores, why vendor numbers run higher, how to run it."
url: https://personalagentbench.com/benchmarks/swe-bench/
last_updated: 2026-11-05
---

# SWE-bench: Verified leaderboard, Pro and what it measures

SWE-bench is a coding benchmark that takes real bug reports from real Python repositories, hands an agent the repository at the commit before the fix, and checks whether the project's own tests pass afterwards. It was [published in October 2023](https://arxiv.org/abs/2310.06770) by Carlos E. Jimenez, John Yang and five co-authors, presented as an oral at ICLR 2024. The SWE-bench team still maintains it, and its [repository](https://github.com/SWE-bench/SWE-bench) lists Jimenez and Yang as contacts, at Princeton and Stanford addresses. It is one of the most cited agent benchmarks, and the clearest example in this section of a benchmark that got its method right and its headline number misread anyway.

- Grading is mechanical. A test suite either passes or it does not, with no judge model and no partial credit.
- The default view on [swebench.com](https://www.swebench.com/) is Verified, 500 tasks checked by human engineers, run for every model in one shared harness. Its leader on 5 November 2026 resolves 76.80%.
- Two entries tie at 75.80% while one costs about five times less per task, and a newer Claude model scores below its predecessor. A rank column shows neither.
- Vendor numbers run higher, up to 79.2% on the same Verified tasks, because they come from the vendors' own scaffolds. SWE-bench Pro is a separate, harder test whose public leader scores 61.50%.
- Nothing it measures involves your accounts, your history or an action that cannot be undone, which is where a personal agent lives.

## What SWE-bench measures, and the variants with the same name

Each instance is a GitHub issue paired with the repository state just before someone fixed it. The agent reads the issue and edits the code. Then the benchmark runs the project's tests, including the ones added with the real fix. Resolved means those tests pass.

That design is the reason this benchmark mattered. The repository's maintainers had already written down what correct means, in executable form, before anyone thought of pointing an agent at it.

The family now has five official sets, each with its own board. The original SWE-bench holds 2,294 instances drawn from 12 Python repositories. Lite is a 300-instance subset for cheaper runs. Verified, [announced on 13 August 2024](https://github.com/SWE-bench/SWE-bench) as the second part of a collaboration with OpenAI's Preparedness team, is 500 problems that software engineers confirmed are solvable, and it is the set most vendors quote. Multilingual is 300 tasks across 42 repositories and 9 languages, and Multimodal is 480 issues described with screenshots or other visual elements. A score quoted without naming its set is close to meaningless, and most quoted scores do not name one.

Several newer benchmarks borrow the name and test something else. [SWE-Bench Pro](https://arxiv.org/abs/2509.16941), from Scale AI, has 1,865 longer problems from 41 repositories, split into a 731-task public set drawn from copyleft repositories, a 276-task private set from 18 startup codebases, and an 858-task held-out set that Scale does not publish results for. [SWE-bench-Live](https://arxiv.org/abs/2505.23419) started with 1,319 tasks from issues created since 2024 and keeps adding fresh ones. [Multi-SWE-bench](https://arxiv.org/abs/2504.02605) covers seven languages other than Python. None of their scores can be compared with the board below.

## The SWE-bench Verified leaderboard

| # | Submission | Harness | Overall | Avg. cost | Dated |
| --- | --- | --- | --- | --- | --- |
| 1 | Claude 4.5 Opus (high) | mini-SWE-agent 2.0.0 | 76.80% | $0.75 | 2026-02-17 |
| 2 | Gemini 3 Flash (high) | mini-SWE-agent 2.0.0 | 75.80% | $0.36 | 2026-02-17 |
| 3 | MiniMax M2.5 (high) | mini-SWE-agent 2.0.0 | 75.80% | $0.07 | 2026-02-17 |
| 4 | Claude 4.6 Opus | mini-SWE-agent 2.0.0 | 75.60% | $0.55 | 2026-02-17 |
| 5 | Claude 4.5 Opus (medium) | mini-SWE-agent 1.16.0 | 74.40% | $0.72 | 2025-11-24 |

As of **2026-10-16** · **47** entries on the bash-only view · [SWE-bench leaderboard](https://www.swebench.com/)

Percentage of the 500 instances in SWE-bench Verified resolved, meaning the repository's own tests pass after the edit. Every row on this view runs in mini-SWE-agent in bash-only mode; the 47 rows span harness versions 0.0.0 to 2.0.0 and the top five use 1.16.0 and 2.0.0, so compare matching versions before attributing a difference to the model. Read from the leaderboard data embedded in swebench.com on 2026-10-16 (earlier reads on 2026-10-08 and 2026-10-12; the top five had not changed). The bash-only view is a filter on the full Verified board, which held 180 entries that day. Avg. cost is the board's mean cost per instance. No confidence intervals are published, so gaps of a point or two cannot be separated from sampling noise.

## Holding the harness still is the real SWE-bench contribution

The default view carries a line that is easy to scroll past: every row is run in [mini-SWE-agent](https://github.com/SWE-agent/mini-swe-agent), a small agent that can only issue bash commands. The site's own tooltip says the point is that "scores compare models rather than harnesses."

That sentence does more work than the scores under it. An agent result is always a product of the model and the scaffolding around it, and most published comparisons change both at once. A vendor reports a number from its own harness, a competitor reports one from theirs, and the difference is credited to the models. Often it is not.

The board goes one step further and prints the harness version beside each entry. On 5 November 2026 the 47 rows ran on versions from 0.0.0 to 2.0.0, and the top five on two of them. Rows from different versions are not strictly comparable, and printing the version is how the board says so rather than hiding it.

## Two SWE-bench entries tie, and one of them costs far less

Second and third place both resolve 75.80% of the Verified set. The average cost per instance is $0.36 for one and $0.07 for the other.

A rank column cannot express that, and it is the single most decision-relevant fact on the board. If you are choosing what to run against a backlog, five times the price for the same resolution rate is the whole question.

The cost column also reframes the top. First place resolves one percentage point more than the two below it, at roughly ten times the cost of the cheaper one. Whether that point is worth buying depends on what the work is worth, which the leaderboard correctly declines to judge for you.

## A newer model scoring lower on SWE-bench is not a mistake

Fourth place, Claude 4.6 Opus, is a later release from the same vendor as first place, and it resolves slightly less of the set.

The reflex is to read that as a regression. It is better read as a reminder of how narrow one benchmark is. A model tuned toward longer tasks or cheaper inference can lose ground on self-contained Python bug fixes while being the better product for most other work.

There are no confidence intervals on this board, which makes gaps of one or two points impossible to interpret. One point on a 500-instance set is five instances. Whether five instances separate two models or two runs of the same model is not something the board lets you check, and it is exactly the question [Terminal-Bench](https://personalagentbench.com/benchmarks/terminal-bench/) answers by printing a margin beside every score.

## The bottom of the SWE-bench board is the more useful half

On 5 November 2026 the 47 entries run from 76.80% down to 9.00%. Almost all of the attention goes to the top five.

The spread says more. The bottom row is Qwen2.5-Coder 32B, released in November 2024. The top row is a model released a year later, and it resolves eight times as many issues on the same fixed harness. That is a stronger statement about progress than any single top score. It also tells you how to read a lone number: a product claiming a resolution rate in the fifties is quoting a figure that sat mid-table a year ago.

Two markings on the board are worth knowing. Fifteen rows are flagged as open-weights models, and thirty are flagged as checked, meaning the benchmark team ran or verified them rather than taking the submitter's word. That second flag is the one to look for. A self-reported number and a reproduced one are different kinds of evidence.

## Vendor SWE-bench scores, SWE-bench Pro and contamination

The bash-only view is a filter on a larger board. Switch it off and the same Verified page lists 180 entries. On 5 November 2026 its top was 79.2%, shared by two entries that wrap Claude 4.5 Opus in their own agent scaffolds, Sonar's and UIUC's live-SWE-agent. The bash-only leader is the same model at 76.80%. That gap is between two configurations; without a controlled comparison it cannot all be credited to the scaffold.

Figures in launch posts usually come from the vendor's own harness, sometimes with several attempts per issue, and the full board tags entries by how many attempts they took. The board has also stopped growing the way it did. On 18 November 2025 the maintainers [changed the submission rules](https://github.com/SWE-bench/experiments): Verified and Multilingual now accept new entries only with an arXiv preprint or technical report and at least one author at an academic institution or research lab, because they want the benchmark kept for research "rather than as a product validation platform." The newest row is dated 26 February 2026, and a commercial product's newer claim will not appear here at all.

SWE-bench Pro is where many newer vendor claims are made instead. On [Scale's leaderboard](https://labs.scale.com/leaderboard/swe_bench_pro), read on 5 November 2026, the public set's leader is Meta's Muse Spark 1.1 at 61.50% ± 3.10, an entry added on 9 July 2026, followed by GPT-5.4 at 59.10% ± 3.56. Both are marked as run in the mini-swe-agent harness, and the board prints a confidence margin beside each score, which the Verified board does not. Pro tasks are longer and the public set is built to resist training-data leakage, so its scores sit well below Verified's. The two numbers should never share a chart.

Then there is the question of whether Verified is still a clean test. [SWE-Bench+](https://arxiv.org/abs/2410.06992) went through passing patches from an early leader by hand and reported that about a third had the solution handed to them in the issue text, and about as many again passed because the tests were too weak to catch a wrong fix. [The SWE-Bench Illusion](https://arxiv.org/abs/2506.12286) found that recent models can name the buggy file from the issue alone up to 76% of the time on Verified, against up to 53% on repositories outside the benchmark, which points to memorisation. Neither paper makes the scores meaningless. Both are reasons to treat a few points at the top as softer than they look.

## Where SWE-bench stops being about personal agents

A SWE-bench instance is a closed problem. The repository is a copy, the tests are the specification, the work is reversible, and when the run ends nothing persists.

A real errand often has requirements that live only in a person's head. Our exam writes selected requirements down as checks on a simulated world: did the flight change, did the message go to the right person, is the subscription actually cancelled. Those checks cover the task as declared, not every preference or consequence in a real account, and the rates on our [scorecard](https://personalagentbench.com/) come from adapted product runtimes rather than end-to-end consumer use. The method is in [how the exam works](https://personalagentbench.com/how-the-exam-works/).

Two ideas from SWE-bench carry over, and we took both. The first is holding the harness constant: every product on our board goes through the same exam environment, or a comparison would measure our integration work instead of the products. The second is grading against state rather than text. SWE-bench asks whether the tests pass, not whether the agent said it fixed the bug, and our exam asks whether the booking exists, not whether the assistant reported making it. An agent that announces a success it did not achieve is invisible to any benchmark that reads only the final message.

What does not carry over is the idea that a clean pass is the whole result. A fix that passes the tests and mangles an unrelated file still counts as resolved here. On a machine that is yours, it would not. The same gap between a versioned score and general ability shows up on [OSWorld](https://personalagentbench.com/benchmarks/osworld/), and [GAIA](https://personalagentbench.com/benchmarks/gaia/) shows a different limit, a crowded top where small differences do not settle a real choice.

## How to run SWE-bench yourself

The harness is open source and runs in Docker, so any published score can in principle be checked. The [repository README](https://github.com/SWE-bench/SWE-bench) gives the current steps: install Docker, clone the repository and run `pip install -e .`, then clone the separate `swe-bench-tasks` repository that the version 5 command line uses to build task images. A quick test that scores the reference fix for one task is `swebench eval verified --gold -i sympy__sympy-20590 --run-id validate-gold --task-repo ./swe-bench-tasks`. To score your own agent's patches, pass a predictions file with `-p`, and `swebench report` re-grades saved logs without rebuilding containers.

Record the dataset, the harness version and the number of attempts beside any result, for the same reason the board does. Lite and Verified are the usual starting points because they are smaller. We did not run a new evaluation for this page.

## FAQ

**What is SWE-bench?** A benchmark that gives an AI agent a real GitHub issue and the repository as it was before the fix, then runs the project's tests to see whether the agent's edit resolves it. It was introduced in October 2023 by Carlos E. Jimenez, John Yang and colleagues, and presented at ICLR 2024.

**What is SWE-bench Verified?** A 500-task subset of the original 2,294, released on 13 August 2024 with OpenAI's Preparedness team, in which software engineers confirmed each problem is solvable. It is the default leaderboard on swebench.com and the set most vendors quote.

**What is the highest SWE-bench Verified score?** On 5 November 2026 the full Verified board's top is 79.2%, using custom scaffolds around Claude 4.5 Opus. On the bash-only view, where every model uses the same harness, the top is Claude 4.5 Opus at 76.80%.

**How is SWE-bench Pro different?** It is a separate benchmark from Scale AI with 1,865 longer tasks from 41 repositories, part of them private or held out. Its public-set leader on 5 November 2026 is Muse Spark 1.1 at 61.50% ± 3.10. Pro and Verified scores are not comparable.

**Why are vendor SWE-bench scores higher than the leaderboard?** Vendors usually report runs in their own scaffolds, sometimes with more than one attempt per issue. The bash-only view fixes the scaffold for everyone, which costs a few points at the top.

**Does a higher SWE-bench score mean a better coding agent?** It means a better result on self-contained Python bug fixes under one harness. Speed, behaviour on long tasks and what the agent damages along the way are either in other columns or not measured.

**Does any of this predict how a personal agent will do?** Not directly. Both involve reading a situation and acting on it, but the failure modes differ. Nothing here tests a standing preference, an irreversible action, or an account that belongs to a person.

## The other benchmark pages

- [**GAIA**Every question arrives cold. Nothing carries over from the last one, so a system that remembers you scores exactly the same as one that doesn't.](https://personalagentbench.com/benchmarks/gaia/)
- [**BrowseComp**Reading, not doing. Nothing is booked, cancelled, paid for or sent.](https://personalagentbench.com/benchmarks/browsecomp/)
- [**WebArena**The sites aren't yours and hold nothing about you. No saved cards, no order history, no account that remembers.](https://personalagentbench.com/benchmarks/webarena/)
- [**OSWorld**Skill, not memory. It asks whether an agent can operate the machine, never whether it knows why you'd want it to.](https://personalagentbench.com/benchmarks/osworld/)
- [**Tau-bench**The user is simulated and the domains are narrow, but this is the closest thing on the list to what a personal agent actually does.](https://personalagentbench.com/benchmarks/tau-bench/)
- [**AgentBench**None of the environments is your life. No calendar, no inbox, no bill to dispute.](https://personalagentbench.com/benchmarks/agentbench/)
- [**Terminal-Bench**Almost nothing a personal agent does happens at a terminal.](https://personalagentbench.com/benchmarks/terminal-bench/)
- [**AssistantBench**Still one-shot. No standing preferences, no routine that fires next Monday.](https://personalagentbench.com/benchmarks/assistantbench/)

No numbers invented. Personal agents taken apart at the runtime level, on a method built to take more.

[See the scorecard](https://personalagentbench.com/)

---

Published by Personal Agent Bench (https://personalagentbench.com), run by the Zentor team. Zentor builds agent software of its own, so its products are not on the scorecard and will not be added.
