Agent benchmarks, and what they miss
There is no shortage of agent benchmarks. Almost none of them measure a personal agent.
SWE-bench asks whether a model can close a GitHub issue. GAIA asks multi-hop research questions. BrowseComp buries a fact and sees who keeps digging. All useful. None of them tells you whether something can book a flight while remembering you refuse red-eyes, or run a routine every Monday without being asked.
The table is the short version: what each one is actually measuring, and the thing it structurally can't see.
| Benchmark | What it measures | What it can't tell you |
|---|---|---|
| GAIA | Assistant-style questions that need several steps, a browser and sometimes a file to answer. | Every question arrives cold. Nothing carries over from the last one, so a system that remembers you scores exactly the same as one that doesn't. |
| BrowseComp | Finding facts that are genuinely hard to find, which mostly tests whether an agent keeps digging. | Reading, not doing. Nothing is booked, cancelled, paid for or sent. |
| WebArena | Completing tasks inside a browser on self-hosted sites, so the environment stays identical between runs. | The sites aren't yours and hold nothing about you. No saved cards, no order history, no account that remembers. |
| OSWorld | Driving a real desktop and its applications rather than a browser tab. | Skill, not memory. It asks whether an agent can operate the machine, never whether it knows why you'd want it to. |
| Tau-bench | Holding to a domain's rules across a conversation with a user, and doing it consistently when the same task is run again. | The user is simulated and the domains are narrow, but this is the closest thing on the list to what a personal agent actually does. |
| AgentBench | One agent across several unrelated environments, which is a decent proxy for general competence. | None of the environments is your life. No calendar, no inbox, no bill to dispute. |
| Terminal-Bench | Work at a command line, where success is unambiguous and cheap to check. | Almost nothing a personal agent does happens at a terminal. |
| AssistantBench | Realistic web tasks that take a while, which is closer to an errand than most of this list. | Still one-shot. No standing preferences, no routine that fires next Monday. |
Pages written so far: GAIA. The rest are in the queue, and each one waits until its leaderboard numbers have been pulled from the source rather than copied off a screenshot.
Why we built a different one
Every row above shares a blind spot: the agent arrives with no history and leaves with none. That's the right call for a research benchmark and the wrong one for judging a product you'd hand your accounts to, which is why this scorecard asks where the model runs, how a tool reaches it and who can read your memory instead.
Start with a teardown, or read how we score.