How results are decided

A result is only useful if you can trust it. Here is how a run decides that a step passed, failed or was blocked.

Passed, failed, blocked

Passed: the step was done and the page shows it worked.
Failed: your site did something wrong, such as an error message, a wrong total or a missing change. This is a finding about your site.
Blocked: the step could not be tested for a reason outside your site’s behaviour, such as the site being unreachable, a missing test account or a CAPTCHA.

A pass is checked against the page

When the agent finishes an act step, it says whether the step worked. That claim is not taken on trust: a separate judge then reads the page and checks it. If the page clearly shows the step did not work, for example “Invalid code” after applying a discount code, the step fails and the run shows both what the agent reported and what the judge saw.

Many steps leave nothing to see (scrolling, opening a menu). The judge only fails a step when the page contradicts it, never because the page shows nothing either way. If the judge cannot give an answer, the agent’s own result stands.

A step repeated from an earlier verified run (replay) is not judged again: the page must match the result recorded then, or the agent works the step out afresh. A replay waits for a slow page about as long as the step took when it was recorded.

Values your site shows for one run only, such as a one-time code, an order number or today’s date, are never repeated from a recording: the agent reads this run’s value from the page. When a recording was made under older rules, for example before an engine update, the step shows Re-recorded with the reason, and it is recorded again.

Checks wait for your page

Results often appear a moment after an action: a report loads, a “Saved” message fades in. A check step does not judge the first thing it sees. It reads the page again every second and passes as soon as the expectation holds. It fails only if the expectation is still not met after 10 seconds.

After every action the agent also waits for your site’s own requests to finish (a save, a search) before reading the page. Requests to other sites, such as analytics, are not waited for.
If something takes longer than 10 seconds, add an act step before the check, for example “Wait until the report has finished loading”.

Browser dialogs

Some sites ask through the browser’s own boxes: “Delete this project?”, “Leave site?”, or a box asking for a name. They are answered OK automatically, the way a person who clicked Delete means it. The run shows each dialog and the answer it got.

To cancel instead, say so in the step: “Click Delete for the Website project, then cancel the confirmation”.
To fill in a box, say what to enter: “Rename the project, entering Roadmap”. When one click opens several boxes in a row (“Place this order?”, then “Enter your gift code”), the answer you give covers all of them.
A dialog is answered as soon as it opens, so a check step after the step that opened it does not see it. Check its effect on the page instead, for example “The Website project is no longer listed”.

New tabs and popups

When a link or button opens a new tab or a popup window (an invoice, “Sign in with Google”), the agent carries on there, as a person would. When a popup closes itself, such as a sign-in window, the agent returns to your page. To go back to an earlier tab, say so: “Go back to the first tab”.

Step screenshots and the live view follow the tab the agent is working in.
The recording shows the first tab only.
A run can have up to 10 tabs open; further ones are closed as they open.

Errors in your site’s code

When your site’s own JavaScript throws an error nobody catches, the agent is told about it. If it follows one of the agent’s actions, the agent reports it as an issue, even when the page looks fine. Errors from other sites’ scripts (analytics, chat widgets) are not reported as issues. Every error, from any script, is in the run’s console log.

Some code catches its own error and only writes it to the console, so a button looks alive but does nothing. The agent is also told about errors your site’s code logs this way. When one follows an action that did not work, it reports it as the likely cause.

Server errors

When a request your page makes to your own server fails with a server error (a 500 after clicking Save, say), the agent is told, and reports it as an issue even if the page shows nothing. Only the request’s path is shown, never its query. Each failing request is reported once.

Things drawn on the page

Seat maps, charts and maps drawn on a canvas have nothing for the agent to read. When looking is turned on, the agent takes a picture of the page, asks a model that can see where things are or what a chart shows, and clicks the point it is given. After signing in, password and code fields are blacked out in the picture, and no picture is taken while a password of the run shows anywhere on the page.

Big pages

On a very large page, the agent reads what matters most first: open dialogs, alerts and messages, then what is on screen, then what is nearest to it. It scrolls to reach the rest.

Digging into a run

Every run page has the recording, a screenshot and the page as the agent read it for each step, and the page’s console and network log. For a closer look, download the run’s Playwright trace from the run page and open it at trace.playwright.dev: it shows every action with the page as it was before and after. The trace is opened in your browser, not uploaded anywhere, and passwords, headers, cookies and request bodies are taken out of it before it is stored. Share links do not include it.

An agent that goes round in circles (the same actions leading back to the same page) is stopped, and the step ends as blocked rather than using up its whole budget.

Tips for reliable results

Follow important actions with a check step that names the visible result. See Writing steps.
Check what a person would see, not how the page is built.
Use a test account and test data you can safely delete, because confirmations are accepted.