Personal Agent Bench

Browser agent failure modes: five focused tests

Browser agent failure modes are the focus of five tasks added to the exam on 4 October 2026, each testing the browser rather than the errand. Each is small on purpose: a code to read off a live screen, a login to recover, a page to check twice 45 minutes apart, an entrance that only exists on a phone, a page that turns away robots. These are the places where real agents get stuck in public, and until now the exam had no task that isolated any of them.

  • Each task targets one browser skill an errand silently depends on. A failure narrows the diagnostic hypothesis; trace review is needed to distinguish model, adapter, infrastructure and grader causes.
  • Every task ships with the wrong answers it must reject: a code read a minute too late, a value reused without rereading, a form filed on the desktop layout.
  • A product states which browser abilities it has. If it doesn't claim one, a task that needs it is filed as "capability not declared under this exam configuration", never guessed.
  • The page that checks for robots is left out of the next baseline. Our own browser can't hide that it's automated, and we won't blame a product for our limitation.
  • No product has a published result on these yet. This page describes the tasks, not anyone's score.

A public browser leaderboard and these tasks answer different questions. WebArena measures complete tasks on self-hosted sites; OSWorld includes desktop workflows; BrowseComp measures difficult lookups. The five tasks here isolate individual browser skills and do not yet have a published product leaderboard. Use the linked benchmark pages for measured results, with their versions and source dates.

Why browser agent failure modes need their own tasks

The method page collects examples such as Xfinity locking Muse out even with browser takeovers, and an AT&T call where the user was patched in once their identity was verified.

An errand task can't tell you which underneath thing broke. If a product fails to book a table, it might have misread the menu, lost its session, run out of time or been blocked as a bot, and the final state looks the same in every case. So the new tasks each strip an errand down to one browser skill, with a small piece of business at the end to prove the skill actually got used. A failure on one of these points at one cause.

That is the difference from the public browser benchmarks. WebArena scores whole tasks on fully working websites from four common domains, OSWorld does the same on a desktop, and BrowseComp asks 1,266 questions whose answers are hard to find on the open web. Each tells you how often the whole job came out right. These five narrow the diagnostic hypothesis; attributing a failure requires trace review to separate model, adapter, infrastructure and grading problems.

They are also deliberately cheap to grade. Each runs in its own small world, uses the same scoring as every other task, and comes with an explicit list of wrong ways to finish that the grading must reject. Those lists are the most useful part of the design, so each section below includes them.

How published browser agent failure modes map to these tasks

Other people have catalogued where browser agents break, and their lists overlap with ours only in part. BrowserArena, a 2025 study that ran models on real open-web tasks and annotated each step, names three consistent failure modes: captcha resolution, pop-up banner removal and direct navigation to URLs. It also reports that DeepSeek-R1 "consistently misleads users about pop-up banner closure", which is a failure of the report, not of the click. WebSuite took the approach closest to ours: a taxonomy of browser actions, with tests of each action alone and in workflows, so that a failed task points to one action. A practitioner post from Browserman argues that production agents fail at handoffs rather than at clicking, and that "a browser session is not just context. It is authority."

Failure mode named elsewhereWhere it shows up here
Captchas and bot checks (BrowserArena)The robot-check task, held out of the next baseline for the reason below
Session and login loss (Browserman)The rebuilt-browser task, which also demands a fresh read after recovery
Handoff to a person (Browserman)The live-code task, graded on reading the code that is showing now
Misreporting what happened (BrowserArena)The live-code task checks the reply and fails a product that names a code it never read
Layout that hides an elementThe phone-only entrance
Stopping before a delayed result arrivesThe 45-minute wait
Pop-up banners, direct URL navigation (BrowserArena)Not isolated by any of the five
Selector drift after a site redesignNot isolated by any of the five

The two gaps at the bottom are real. An agent that cannot close a cookie banner fails errands on this exam too, but nothing here tells you that was the cause.

Failure mode 1: reading a live code during a handoff

The task. A verification step shows a one-time code on a live screen. The agent has to open the live view, read the code that is showing now, enter it, and then finish the business step that the verification unlocks.

This is the shape of every handover that involves a person: the agent can get you to the door, and something about the door needs a fresh reading. The interesting failures are about freshness, not about reading.

Wrong ways to finish, all graded as not reaching the goal:

  • Read the code, wait a minute, then enter it. The screen has moved on and the code is stale.
  • Enter a code without ever opening the live view. Even if the digits happen to be right, the agent didn't look.
  • Do nothing.

The reply is also checked: the product has to say which code it saw and what happened, and must not invent a code it never read.

Failure mode 2: losing the login when the browser is rebuilt

The task. The agent logs in to a members' site, looks up the most recent renewal order number, and files it on the site. Partway through, the browser is shut down and rebuilt from nothing. The agent is told this may happen and is expected to log back in, read the order number again, and only then file it.

Real browsers die. Sessions expire, sandboxes get recycled, a long task outlives the machine it started on. What matters is whether the agent notices that its picture of the world went stale when the browser did.

Wrong ways to finish:

  • Reuse the order number it read before the rebuild without reading it again. The number happens to be the same here, and it still fails, because the task requires a fresh observation after the rebuild, even when the value is unchanged.
  • Restore the browser without logging in again, then try to carry on.
  • Do nothing.

The first of those is the one worth dwelling on. A grader that only checked the final value would pass it. This one checks that the value was read after the rebuild, which is the difference between an agent that recovered and one that got lucky.

Failure mode 3, the hardest: holding one browser for 45 minutes

The task. A process takes 45 minutes to produce a result. The agent has to read a page at the start, read it again at least 45 minutes later in the same browser, and then file the confirmation number that appears.

This is the "keep an eye on it for me" task in its barest form, and it's the one we expect to be hardest. Our September rehearsal notes describe watch-later tasks failing when agents declared the job done before the event arrived: the agent ended its work as finished before the event it was meant to wait for. That observation has not been published with its counts, so read it as a working note rather than a measurement. This task asks the underlying question directly: can the product keep a browser alive and come back to it?

Wrong ways to finish:

  • Read twice, but only a minute apart.
  • File the confirmation after the first read, without waiting for the second.
  • Do nothing.

One variation is explicitly allowed: doing everything a second late still passes, and causes no damage. A grader that punishes a one-second slip is measuring the clock, not the agent. While a product is working, the exam's clock runs at real speed, so the 45 minutes are real minutes.

Failure mode 4: an entrance that only exists on a phone

The task. The user says they're on their phone and want to book through a mobile-only reservation channel. The channel only appears in the mobile layout. The agent has to switch to a phone-sized view, find the entrance and file the booking through it.

Sites increasingly hide things behind layout: a phone-only offer, a desktop-only export, a button that moves when the window narrows. An agent that only ever sees the desktop layout will tell you the option doesn't exist, and from where it's standing, it's right.

Wrong ways to finish:

  • File the booking through the desktop path.
  • Switch to the phone layout but book through an entrance that isn't the mobile one.
  • Do nothing.

The exact screen size is the product's choice. A different phone size that still produces the mobile layout passes. The task cares about reaching the mobile entrance, not about matching a particular device.

Failure mode 5: a page that checks whether you're a robot

The task. A partner's public price page runs a check for automated browsers before it shows anything. The agent has to get through it, read the monthly renewal quote, record it on the site and report it.

Wrong ways to finish:

  • Record the value without opening the page at all.
  • Read the page but record a number off by one.
  • Get through with an ordinary automated browser that announces itself as automated. The page sees it, and the quote is never legitimately read.
  • Do nothing.

This task is real and it is graded like the others, but it is not in the next baseline batch. The reason is ours. When the exam supplies the browser itself, that browser can't disguise the fact that it's automated, so a product using our browser would fail this task for a reason that has nothing to do with the product. We left it out rather than record our own limitation as someone else's failure. Products that bring a browser able to pass the check can still be run on it separately, and any such result will be labelled as such. What site owners can and can't conclude in the meantime is on is your website agent ready?

A product says what its browser agent can do, and is held to it

Each product connected to the exam publishes a short description of itself, and part of that description is which of three browser abilities it has: getting past robot checks, changing the screen size, and keeping one browser alive over a long wait. Logging back in after a rebuild and handing a live screen to a person are declared the same way, through the parts of the description that cover sessions and takeovers.

The rule is strict in both directions. Only an explicit yes counts. A missing answer or a no means the capability is undeclared under this exam protocol, not that the real product necessarily lacks it, and a task that needs it is filed as "capability not declared under this exam configuration". It is never inferred from a product's name, its marketing, or whether its browser is its own. And a product that claims an ability it doesn't have will fail the task on the evidence, because the task checks what actually happened.

That split matters for reading results. "Can't do this" and "got it wrong" are different findings, and on this site they're always reported as different things. A product that honestly declares it can't hold a browser for 45 minutes has told you something true about itself, which is more than a product that claims it can and then quits after one read.

What these browser tasks can't tell you yet

There are no results on this page, because none have been published. The next baseline batch is drawn to run each task twice per product, and when results land each one will be reported with its sample size, under the rules it was run with.

Two limits are worth knowing in advance. First, our own browser can't be resized after it starts, so the phone task depends on a product being able to start in, or switch to, a mobile layout some other way, and any comparison has to say where that mattered. Second, a task this small is a test of one skill, not of a whole errand. Passing all five doesn't mean a product can do your errands. Failing one tells you where it will probably break.

FAQ

Why only five browser tasks? Because each one isolates a single skill, and these five cover the patterns we chose to start with: stale handovers, lost sessions, long waits, layout-gated options and bot checks. We have not measured how often each one occurs, so the choice is not a frequency ranking. Four of the five are in the next baseline batch of 169 tasks, each run twice per product; the robot-check task is the one left out. More can be added, and when they are they'll be recorded on what changed in the exam with their dates.

Is getting past a bot check something an agent should do? The task uses a public price page that the user is entitled to read, and the agent records a public number. It tests whether an agent can do on the user's behalf what the user could do by hand. It is also the one task we left out of the baseline, for the reason explained above.

Does a product fail if it doesn't declare a browser ability? It's filed as "capability not declared under this exam configuration", which is reported separately from getting a task wrong. An undeclared capability and a failed attempt are different exam statuses; neither alone proves that the shipping product lacks an ability.

Are the 45 minutes real? Yes, while a product is working the exam's clock runs at real speed. Only our own grading checks fast-forward through the wait, and those test the grader, not the product.

Where these browser tasks sit in the exam

The rest of the exam is described on how the exam works, and the date these tasks were added is on what changed in the exam. How each task's wrong answers are checked against the grading code is on how we check our own grading. The errands these skills sit underneath, such as booking a table or live alerts, have pages of their own. For errand attempts that were actually recorded, sorted by what the agent did wrong rather than by which browser skill broke, see AI agent failure modes, a separate page with the counts and the attempts themselves.