GAIA benchmark: what it measures, and why the top score passed the humans

GAIA asks questions a person could answer given an afternoon, a browser and some patience. That was the whole idea, and it's the reason the benchmark mattered.

What it actually is

466 questions, built so that the hard part is the work rather than the cleverness. Most need several steps: search for something, open a file, read a table, follow a link, come back with one exact answer. The answers to 300 of them were never published, which is what keeps the leaderboard honest.

The design goal was stated plainly in the paper, and it's worth quoting the shape of it rather than the words: rather than chase problems that are hard for people, GAIA went after problems that are easy for people and hard for machines. Humans scored 92%. GPT-4 with plugins managed 15%.

That 77-point gap was the point of the whole thing.

Where the leaderboard stands

The gap is gone.

Read the top line again: 93.4%, against a human baseline of 92%. Level 3 questions, the ones meant to be hardest, are being answered nine times in ten.

We pulled every submission in the public results set through the dataset API and sorted it ourselves rather than reading numbers off a rendered page, because a leaderboard screenshot is not a source. The figures above are what that pull returned on 21 September 2026, and you can run it yourself.

What GAIA can't tell you

Every question arrives cold.

That one sentence is the whole limitation, and it's structural rather than fixable. A GAIA run has no yesterday. The system answering question 300 knows nothing it didn't know at question 1, so a product that remembers you scores exactly the same as one that forgets you the moment you close the tab. Personalisation is worth zero points. So is a standing preference, a routine that fires on Monday, a correction you made last week that should stick.

There's a second gap that matters more for anything you'd actually hand your accounts to: GAIA is read-only. Every answer is a fact retrieved. Nothing gets booked, cancelled, disputed or sent, so nothing can go wrong in the way things go wrong when an agent acts on your behalf. No wrong account gets charged. No call gets placed to the wrong company.

And now saturation. When the top of a leaderboard sits above the human baseline the paper set as its target, the benchmark has stopped separating the top contenders from each other. It still tells you a system is competent. It no longer tells you which one to use.

So what would tell you

The questions GAIA can't ask are the ones that decide whether a personal agent is any good: where does the model actually run, what can it reach on your machine, how does a tool get in front of it, and can you read — or correct — what it believes about you?

Those are the axes on our scorecard, built from taking four of these products apart at the runtime level rather than from scoring them on quiz questions. It carries no numbers yet, and it says so; the numbers come from recorded task runs, and inventing them would make it the thing this page just criticised.

Read how we score, or start with the Pine teardown.