Agent benchmark changelog: what changed in the exam, and why
This agent benchmark changelog records changes to the rules behind the scorecard, including how outcomes are classified, how unwanted actions are counted, and how the simulated user responds. It distinguishes a new rule from a correction to a published fact. The September completion figures were measured on 22 September 2026 and have not been rescored under the later changes below. The October batches, run between 11 and 14 October, came after the 28 September rules; we do not have the exact exam revision of each, so we do not say which later changes each one includes. Last change recorded: 10 October. The exam repository was last compared on 12 October 2026, and no later comparison has been recorded.
- September completion rates retain the rules they were measured under; new and old measurements are not averaged.
- Goal completion, reply truthfulness and unwanted actions remain separate measurements, with no combined score.
- Questions the task author did not anticipate no longer automatically void a run.
- The exam now delivers urgent and noise events to products, rather than making them discover every event by polling.
- Failures in the exam's own clock, sandbox and simulated user are assigned to the exam under the rules below.
| Date | Change | Effect on scores |
|---|---|---|
| 28 September | Outcome classes and action counts | Changes which failures remain in the product record |
| 28 September | Questions and reply grading | Keeps unanticipated questions in the sample; adds separate reply score |
| 3 October | One adapter protocol | Reduces product-specific adapter differences |
| 4 October | Browser tasks and fixed draw | Changes coverage and fixes selection before results |
| 4 October | Versioned records | Prevents combining incompatible rule versions |
| 10 October | Exam faults and user handling | Removes some exam-caused failures from product results |
| 10 October | Forms, events and model route | Removes duplicate-write traps; tests response to delivered events |
What this agent benchmark changelog covers
The exam repository is not public. This page describes the snapshot checked on 12 October 2026: scoring contract version 5, repository revision cfd6d38e (an internal reference, not a publicly inspectable contract diff), recorded in the site's exam-watch data. That identifies the record being described; it does not make the private repository available for independent inspection.
We compare the scoring version, the document explaining how results are read, the task library and the batch task lists. The check is manual for now. Changes confined to code can escape that comparison, so this is a record of known rule changes, not a guarantee that every implementation change is listed. Corrections to the site's published claims belong on the corrections page.
The September figures on the scorecard and the September smoke run retain their September dates and original rules. Six October batches, run between 11 and 14 October, are published on the results page under their own dates and never pooled, with their attempts on the recorded runs pages; the exam team says each used different tasks, exam code and product builds. We do not have the exact repository revision of each October batch, so we do not say which of the changes on this page each one includes. The current method page describes the later exam. Reading a September attempt under today's event-delivery rule, for example, would assign it a capability that the old task never exercised. The next published batch must identify its own version and measurement date.
28 September: new scoring rules
Six outcomes became five
Before: the classes were succeeded, wrong final state, product lacks a capability, timed out, exam broke, and environment cannot run the task. The last two did not count against the product.
After: the environment-only class was removed. When a task needs a capability the environment does not declare, the run is classified as a missing product capability and remains in its record. A broken exam still does not count against the product.
The change record gives no reason for that reclassification. Its consequence is nevertheless clear: an environment gap can now cost the product a mark. Required task capabilities and declared environment capabilities are frozen with the batch, so anyone holding the record can inspect the basis for that decision. The published figures have not been recalculated under it.
Early and late actions became one measurement
Previously, acting before an event and acting after a deadline were separate headline counts. From 28 September they share the mistimed-action measurement. Each run still keeps early and late tallies and the time of each attempt. Combining the headline does not erase which side of the permitted window the product missed.
Unauthorised actions have one line
Payment attempts previously had different names in shopping and travel and were not counted uniformly across worlds. Signatures and final submissions were mixed into early actions. The new unauthorised-action count covers payments, signatures, final submissions, outward messages and logins without permission across all worlds.
On 2 October, writing to an account without permission moved here from invalid operations. On 4 October, an unrequested outward effect joined it: an extra form, an alert about nothing, or a message to someone outside the requested recipients. Drafts and local files do not count as outward effects because they have not been sent.
An alert about nothing had briefly been counted as damage. The new placement distinguishes changing or breaking the user's state from sending something in their name that they did not request. Neither count cancels a successful goal; both are printed separately.
28 September: questions and reply scoring rules
An unanticipated question no longer voids a run
Previously, asking the simulated user a question the task had not declared caused the run to be discarded as a broken exam. A 28 September dinner-task smoke run exposed the problem: asking which Friday was meant was sensible, but removed the whole attempt from the sample.
Now an undeclared question receives a fixed response saying no additional information is available, and the run continues. A declared question with a missing answer is still an exam fault. This prevents products that ask many questions from disappearing disproportionately from the measured sample, while keeping the simulated user from inventing facts.
Every task also gets a reply score
Previously, the grader inspected the world's state without reading the final reply. Written work could be held out until a person reviewed it. From 28 September a model judge gives a separate score from 0 to 10 for truthful reporting and any reply requirements declared by the task. Structured goals still depend on world state; a convincing reply cannot manufacture a successful booking or cancellation.
The judge takes three samples and uses their median. That followed instability in the first trial: a single judgment ranged from 3 to 8 on 13 of 32 runs. All three samples must succeed, or the review is an exam-side judge failure. The October retry rule below reduces malformed responses without changing that requirement.
The judge receives the facts read by the task's standard solution on a fresh world, consistently across products. It does not receive a more favourable evidence packet merely because one product browsed more pages. A claim is treated as fabricated only when recorded world actions contradict it; absence of a browsing action from the record does not prove the product never browsed.
3 October: the exam replaces hand-written adapters
Before this change, each product had a separate hand-written adapter. From 3 October one protocol connects products to the exam. A product supplies a description and a container recipe; each run starts a fresh sandbox from that recipe. Model and browser credentials remain outside the product in a proxy created for the run.
This matters because adapter faults can look like product faults. The earlier Muse result changed after four defects in our adapter were fixed, as the scorecard records. One connection protocol reduces the amount of product-specific glue that must be checked. It does not prove that the shared implementation is fault-free, and the later sandbox and conversation corrections show why that distinction matters.
4 October: browser tasks and the exam's fixed draw
Five small browser tasks were added across four worlds: reading a code from a live screen, restoring a login after rebuilding the browser, maintaining a browser through a 45-minute wait, using an entrance available only in a phone layout, and passing a robot-check page. The login-restoration and mobile-layout tasks share one world. These isolate capabilities that a longer errand can fail for several different reasons.
Products declare their capabilities. A task requiring an undeclared capability is classified as beyond the product. The robot-check task was excluded from the baseline because the exam's supplied browser cannot conceal its automation. The browser task descriptions explain the wrong answers each task must reject.
The planned baseline contains 169 tasks, chosen by grouping tasks by the change they require, allowing at most two per group, and choosing a second task to cover a different difficulty. A fixed seed and no inspection of product results constrain the selection.
There is an unresolved historical count. The earlier draw notes describe three real-account exclusions, but the current library lists five: Google Calendar self-event, Notion self-page, self-only calendar read, self-only calendar write and Slack self-message. The supplied record does not identify the three excluded then. We therefore do not claim that three is the current inventory or silently recalculate the historical draw. How tasks are drawn preserves the recorded counts with this limitation.
Every batch and run also carries its scoring-rule version. The current contract is version 5. Incompatible versions are not read or averaged together, and old evidence is not converted into a new result format. The Terminal-Bench version discussion provides another example of why the exam version belongs beside a score.
10 October: exam failures and simulated-user rules
The changes made on 9 and 10 October entered the main exam revision on 12 October. They did not change contract version 5. Consequently the contract number alone cannot distinguish runs immediately before and after them; the exact repository revision matters. The October batches on the results page ran between 11 and 14 October, on both sides of that date, and we do not have the revision each one ran; read them as separate measurements, as they are published.
More infrastructure faults belong to the exam
Previously, clock failure during a run and sandbox failure after the first message could be charged to the product. Now sandbox creation, access, continued operation and teardown failures, plus clock failures, are exam faults at any point.
An error exit from a command inside the sandbox remains product-side by default: the exit code alone cannot identify whether the exam command or product state caused it. Conversation faults caused by the simulated user, such as sending text where a form requires an option, are exam faults. A 9 October preflight also checks that the product accepts the exam site's security certificate under the simulated date.
Reclassified faults leave the product's scored record. They still contribute to the check that refuses a batch when more than one run in five is a broken exam. A lower product failure count after this change may reflect attribution, not better product behaviour.
Malformed judge responses get one retry
A malformed sample from the reply judge is now requested once more before it fails. The explanation can use recognised alternative field names, but the score must remain an integer from 0 to 10. All three samples must still succeed and the final score remains their median. This reduces missing reviews; it does not establish that every retry leaves an individual score unchanged.
Questions, forms and refusals are handled more carefully
The exam model can match a reworded question to a task-declared question; it does not invent the answer. Tasks may opt out and require exact declared wording. The simulated user replies when a declared question is asked or the product is waiting, rather than responding to every question mark and overwriting a finished report.
Declining approval now withholds permission without cancelling prepared work. A keep-as-is option is preferred when available. If nothing fits, the form is closed with a message that nothing was selected and nothing should change. Fields are handled individually so one unanswered field does not discard valid answers elsewhere. These repairs can improve goal results by stopping the exam's stand-in user from undoing correct work.
10 October: exam delivery, events and model routing
Forms no longer provoke repeated writes
The review found successful writes followed by error or empty pages, button-only forms that never reached the world, and email addresses rewritten by the tunnel. Those faults could make an agent retry a submission that already succeeded.
Requests now reach the world and leave records; saved forms display their result, navigation is labelled consistently, and pages leave the exam unchanged. Duplicate side effects remain a product measurement, but older repeats cannot all be cleanly separated into product and exam causes. They must not be used as if the repair had already been in place.
Urgent and noise events are pushed to products
Previously, events appeared inside the world without a delivery path to the product, and an alert counted only after the product read the event itself. Those tasks measured polling as well as deciding when to speak.
Now events are delivered in occurrence order without revealing whether the user wants an alert. Products must declare an event receiver; without one, these tasks are classified as beyond their capability. Responding to an unwanted event counts as an unauthorised outward action. Deadlines still start at event time, and the exam clock runs at real speed for up to five minutes afterward so model-call latency is not outrun by fast-forwarding.
Products share a model route
The exam already substituted its model for the model requested by each product, but product routing preferences could still select the provider. The new rule removes those preferences and allows the product to receive the exam model through a declared setting. This reduces provider differences being mistaken for product differences. It is a control on the comparison, not a claim that the replicas reproduce every production configuration.
For external context, the BenchJack paper documents how grader defects can distort scores. Terminal-Bench’s releases distinguish task sets by version. Our version record serves the same practical need: a changed exam must be identified before its numbers are compared.
The reported September comparison used replicas with the same GLM 5.3 flash model substituted, so it establishes nothing about the shipping products' own models; the results page sets out that boundary. Runtime adaptation does not establish exact fidelity to every shipping feature. Supplied common-batch transcripts are not public and include overwritten or shared files; the rates are not end-to-end consumer-product measurements.
11 to 14 October: retries, the exam's fixed draw and grading fixes
The fixed draw named a renamed task
When the long browser task was shortened to a 45-minute wait on 4 October, its file was renamed, but the fixed baseline list kept the old name. A formal baseline batch using that list could not start. The list was corrected on 11 October. The draw itself did not change, and because a batch using the broken list could not start, no published figure came from it.
A retried attempt no longer splits a batch
Previously, when a shared service failed and one attempt was run again, the exam lowered the number of attempts running at once for the rest of the batch and filed later attempts as a separate group. A batch with two groups was refused outright, so a single retry could throw away a whole batch. From 11 October that number is fixed for each batch, a retried attempt counts as an independent attempt under the same conditions, and every retry is listed in the batch record. The voided attempt is no longer discarded either: its full record is kept as evidence and never enters a score.
Exam-side faults stopped counting against products
On 14 October, after going back over earlier records, several faults were moved from the product's account to the exam's:
- A slow health check on the exam's own cloud sandbox could mark a whole attempt as a broken exam. A failed reading now fails only that reading, and the existing rule of three consecutive failures decides.
- A request for a page that does not exist now gets a plain "not found" on every simulated site. On some sites it used to return an error that counted as an invalid action.
- A message the errand itself required, worded within what the task allowed, is no longer counted as an unrequested action.
- When the time limit is reached, a reply the product had already delivered is kept even if background work it started is still open. Such attempts used to score 0.
Sixty Gmail tasks from a public benchmark
On 14 October the exam imported 60 tasks from the Gmail world of WebArena Infinity, a public benchmark. They are not in the fixed draw. Parts of the score that this world cannot observe, such as timing and unrequested actions, are recorded as not measured rather than as zero, in line with the rule that a missing measurement never becomes zero.
These changes apply to attempts graded after they landed. September completion rates keep the rules they were measured under and are not recalculated.
FAQ
Did these changes move the published scores? Not the September ones. They keep their original dates and rules and were not rescored. The October batches were run after the 28 September rules took effect, so they are published as separate measurements and are not comparable with September. We do not know each October batch's exact exam revision, so we do not say which of the 9 and 10 October changes each one includes. A future batch must identify its version before readers compare it with earlier measurements.
Why not rescore September runs? Replaying old evidence under new rules would attach a different exam's verdict to it. Keeping the original interpretation makes the historical record clearer and avoids a falsely continuous series.
Does reply grading mean trusting what an agent says? No. Structured goals are judged from the world. The reply score measures truthful reporting separately; it cannot replace a missing state change.
Where are the current rules? Read how the exam works for the current method and how we score for the teardown axes. This changelog records the transitions and their limits.