Every browser automation tool has the same marketing line: fast, cheap, reliable, one line to install. None of them are compared on the same job under the same conditions, so "fastest" is usually whatever the author measured on their own laptop.
I wanted the boring, comparable version. So I built web-automation-microbench: one fixed job, one browser situation, one pass test that reads the page independently of whatever the agent claims. Then I ran 33 tools against it and put them on one leaderboard. The full leaderboard is here.
One job, one honest pass test
The task is deliberately small: on the public TodoMVC demo app, add two to-dos ("Email supplier", "Review invoice"), tick off only the first, switch to the Active filter, and confirm only "Review invoice" is left with "1 item left".
Two rules keep it honest:
- Pass or fail is decided by code, not by the agent. A separate checker reads the page and the app's saved data. If a tool says "done" but the todo was never committed, it fails. That mattered more than I expected.
- Failed runs are kept. Nothing is retried away to make a median look better.
The stopwatch covers only the model-plus-browser work — first model call to "done" — not browser startup or the checking step.
What actually won
Thin harnesses that let the model issue one native browser command per step won on both speed and cost. The top of the table is extension-backed CLIs finishing in about four seconds for half a cent:
| Harness | Median | Tokens in/out | Cost per run |
|---|---|---|---|
| BrowserSkill | 4.1s | ~6.2k / ~97 | ~$0.0005 |
| browser-relay | 4.3s | ~5.9k / ~95 | ~$0.0005 |
| browser-cli | 6.2s | ~8.6k / ~128 | ~$0.0007 |
| Browser Harness | 9.9s | ~6.2k / ~190 | ~$0.0005 |
The pattern is not "which company", it is "how much ceremony sits between the model and the browser". Anything that runs its own heavyweight agent loop — BrowserCode, notte, skyvern, midscene — landed at 33 to 187 seconds for no accuracy gain. midscene burned roughly 444k input tokens per run doing it. Once you see that on one screen, the choice gets easier.
The part that surprised me
A leaderboard sorted by time tells you almost nothing about whether a tool can do real work. So I scored the same harnesses a second time on 11 real tasks pulled from my own recorded sessions — tag inspection, script inventory, SEO audits, crawlability checks, and consent-to-add-to-cart journeys on live retail sites.
The two rankings disagree, hard. BrowserSkill is the fastest tool on the microbenchmark and among the weakest on real work. browser-relay is a hair slower on the toy task and the strongest on real work. If I had only published the speed table, I would have recommended the wrong tool.
The difficulty split is clean, too: one-shot "inspect the live page and report" audits passed on almost anything that can evaluate JavaScript (8 out of 9 harnesses). The multi-step journeys — consent, find the product, add to cart, verify the tags — passed 0 to 2 times out of 9. Complexity, not harness quality, is what broke them.
Three failures I did not expect
- page-agent ships no key-press action at all, so it can never press Enter to commit a todo. It filled the field across roughly 16 attempts on 5 tabs and still reported failure. I only got it working after patching in
send_keysand dispatching the key from the page's main world. - opencli and bb-browser reach the page and claim success, but their key event carries no
keyCode, so React never commits the todo. Two different projects, same upstream defect. I confirmed the root cause with a direct probe. - Every extension-based tool kept failing to load until I realised branded Chrome silently ignores
--load-extension. Chrome for Testing honours it. One line of setup unlocked five tools I had wrongly written off.
How to use it
If your job is "open a page, read something, report it", pick by speed and cost — the top rows are all fine, and they are cheap enough that the difference is noise.
If your job is a multi-step authenticated journey, do not pick from the speed table at all. Screen your real tasks against two or three candidates, run more than two reps, and watch for the false "done" — the pass test caught several of those here.
What this does not prove
TodoMVC is a controlled instrument, not a proxy for real work, and I treat the two tables as separate claims. Several rows are two-rep screening results and should not be ordered finely against each other. Models matter as much as harnesses, so a different model could move the middle of the table. And because I wrote both the tasks and the test, the safest thing you can do is re-run it yourself — the repo has the exact task text, the pass rules, and the raw per-run JSON.
That is the whole point: not "trust my leaderboard", but "here is a fair race you can rerun". The leaderboard lives on /solutions, and the repo is the source of truth behind it.