Solutions · Benchmarks
Web automation tool leaderboard
33 browser automation tools, one identical job, one independent pass test. Two rankings: fast-path speed and cost on a controlled microbenchmark, and real-work capability on 11 harvested tasks. They disagree, and that is the point.
The job every tool had to do
On the public TodoMVC demo app: add two to-dos (“Email supplier” and “Review invoice”), tick off only the first, switch to the Active filter, and confirm only “Review invoice” is showing with “1 item left”.
Passing is checked by independent code reading the page and the app’s saved data — never by the agent’s own say-so. The stopwatch covers only the model-plus-browser work, and failed runs are kept, never retried away.
Fast-path leaderboard
One row per harness and best-model combination, sorted by median time. Thin harnesses that let the model issue one native command per step win on speed and cost; heavyweight agent runtimes cost 10–100× more wall-clock for no accuracy gain. Rows marked scored failure reached the page but could not finish the job.
| # | Harness | Pass | Median | Tokens in/out | Cost / run |
|---|---|---|---|---|---|
| 1 | BrowserSkill Tencent/BrowserSkill · round 3 | 2/2 | 4.1s | ~6.2k / ~97 | ~$0.0005 |
| 2 | browser-relay reliefeai/browser-relay · round 3 | 2/2 | 4.3s | ~5.9k / ~95 | ~$0.0005 |
| 3 | browser-control keon/browser-control · round 3 | 1/2 | 5.5s | ~13.2k / ~180 | ~$0.0007 |
| 4 | browser-cli six-ddc/browser-cli · round 3 | 2/2 | 6.2s | ~8.6k / ~128 | ~$0.0007 |
| 5 | pinchtab pinchtab/pinchtab · round 3 | 2/2 | 8.3s | ~5.1k / ~200 | ~$0.0003 |
| 6 | browser-use browser-use/browser-use · round 1 Round 1 timing-only (tokens not recorded) | 2/2 | 8.7s | n/a | n/a |
| 7 | Browser Harness browser-use/browser-harness · round 2 | 2/2 | 9.9s | ~6.2k / ~190 | ~$0.0005 |
| 8 | agent-browser vercel-labs/agent-browser · round 3 | 2/2 | 10.1s | ~20.8k / ~120 | ~$0.0010 |
| 9 | Playwriter remorses/playwriter · round 1 Round 1 timing-only (tokens not recorded) | 2/2 | 10.1s | n/a | n/a |
| 10 | cdp-browser sids/cdp-browser · round 3 | 2/2 | 10.7s | ~6.3k / ~250 | ~$0.0003 |
| 11 | jarvis-browser bridge25/jarvis-browser · round 3 | 2/2 | 11.2s | ~8.5k / ~100 | ~$0.0005 |
| 12 | chrome-cdp-skill pasky/chrome-cdp-skill · round 6 Fastest new arrival, least reliable once promoted to 5 reps | 3/5 | 11.5s | ~21.1k / ~323 | ~$0.0083 |
| 13 | chrome-devtools-mcp ChromeDevTools/chrome-devtools-mcp · round 6 Strongest Round 6 row once reliability is counted | 5/5 | 11.8s | ~12.6k / ~89 | ~$0.0048 |
| 14 | Stagehand v4 browserbase/stagehand · round 2 | 1/4 | 13.1s | ~6.6k / 300–3,400 | ~$0.0007 |
| 15 | ego-browser citrolabs/ego-lite · round 6 | 5/5 | 14.2s | ~12.3k / ~99 | ~$0.0048 |
| 16 | webctl cosinusalpha/webctl · round 3 | 2/2 | 14.5s | ~8.2k / ~100 | ~$0.0006 |
| 17 | playwright-cli microsoft/playwright-cli · round 6 | 5/5 | 14.7s | ~21.7k / ~113 | ~$0.0083 |
| 18 | agent-chrome-cli gxbvc/agent-chrome-cli · round 3 | 2/2 | 17.4s | ~7.6k / ~180 | ~$0.0004 |
| 19 | browser-act-skills browser-act/skills · round 6 | 2/2 | 18.5s | ~6.0k / ~76 | ~$0.0009 |
| 20 | lightpanda lightpanda-io/browser · round 3 | 2/2 | 22.7s | ~7.1k / ~590 | ~$0.0006 |
| 21 | page-agent (patched: send_keys) alibaba/page-agent · round 5 Patched build; stock ships no key-press action and scores 0/2 | 2/2 | 25.3s | n/a | n/a |
| 22 | hyperagent (perform) hyperbrowserai/HyperAgent · round 6 | 3/4 | 28.3s | ~20.8k / ~3.4k | ~$0.0096 |
| 23 | opencli jackwener/OpenCLI · round 6 Scored failure: key event carries no keyCode, so React never commits the todo | 0/2 | 30.4s | ~20.8k / ~234 | ~$0.0032 |
| 24 | surf-cli nicobailon/surf-cli · round 6 | 2/2 | 31.8s | ~28.6k / ~519 | ~$0.0046 |
| 25 | browser-agent visnia-ai/browser-agent · round 4 | 2/2 | 32.5s | ~43.1k / ~1,360 | ~$0.0036 |
| 26 | bb-browser epiral/bb-browser · round 6 Scored failure: native key event carries no keyCode | 0/2 | 32.7s | ~12.6k / ~380 | ~$0.0021 |
| 27 | midscene web-infra-dev/midscene · round 6 Vision-first; belongs to the capability suite, not the speed ranking | 2/2 | 33.4s | ~444k / ~62k | ~$0.074 |
| 28 | raw-playwright baseline microsoft/playwright · round 3 | 3/4 | 42.7s | ~18.1k / ~410 | ~$0.0010 |
| 29 | Magnitude magnitudedev/magnitude · round 2 Vision-first: most reliable on messy JS sites, slower and pricier | 4/4 | 52.6s | ~18.3k / ~2.9k | ~$0.0021 |
| 30 | BrowserCode uuuuytgg/browser-code · round 2 Own heavyweight agent loop; 10–100x more wall-clock for no accuracy gain | 2/2 | 153.0s | ~55.6k / ~3.0k | ~$0.026 |
| 31 | notte nottelabs/notte · round 3 | 2/2 | 171.8s | n/a | n/a |
| 32 | skyvern Skyvern-AI/skyvern · round 6 Autonomous multi-agent; capability suite | 2/2 | 186.5s | ~29.0k / ~4.2k | ~$0.0055 |
| 33 | browser-agent (Taylor-Bayouth) Taylor-Bayouth/browser-agent · round 5 Unrelated project that shares the visnia-ai name | 1/2 | 200.1s | ~19k–782k / ~0.5k–25k | n/a |
Real-work capability leaderboard
The same harness set scored on 11 real, provenance-backed browser tasks harvested from recorded sessions — tag inspection, script inventory, SEO audits, crawlability, canvas creation, and consent-to-add-to-cart journeys. The latency ranking and the capability ranking genuinely disagree: the fastest tool is the weakest on real work.
| Harness | Fast-path | Real-work | Reps |
|---|---|---|---|
| browser-relay reliefeai/browser-relay Strongest on real work; promoted past screening | 2/2 · 4.3s | 22/33 | 1, 2, 3 |
| raw-playwright baseline microsoft/playwright Promoted past screening | 3/4 · 42.7s | 17/33 | 1, 2, 3 |
| agent-browser vercel-labs/agent-browser | 2/2 · 10.1s | 6/11 | 1 |
| cdp-browser sids/cdp-browser | 2/2 · 10.7s | 6/11 | 1 |
| BrowserSkill Tencent/BrowserSkill Fastest fast-path, weakest real work — the orderings invert | 2/2 · 4.1s | 2/11 | 1 |
Per-task difficulty
Pass counts across all harnesses. One-shot “inspect the live page and report” audits converge on almost any harness that can evaluate JavaScript; multi-step journeys and canvas construction sit above the current frontier.
| Task | Capability | Passes |
|---|---|---|
| allbirds-uk-add-to-cart-tag-check | consent → add to cart → tags | 0/9 |
| gymshark-uk-add-to-cart-tag-check | consent → add to cart → tags | 1/9 |
| puma-uk-seo-metadata-audit | SEO / structured data | 1/9 |
| chanel-gb-pdp-tag-inspection | tag inspection (anti-bot boundary) | 2/9 |
| tldraw-three-shape-diagram | canvas UI creation | 2/9 |
| rajeevg-crawlability-audit | robots.txt + sitemap | 7/9 |
| porsche-uk-script-inventory | third-party script inventory | 8/9 |
| porsche-uk-tag-inspection | tag inspection | 8/9 |
| puma-uk-script-inventory | third-party script inventory | 8/9 |
| puma-uk-tag-inspection | tag inspection | 8/9 |
| rajeevg-seo-metadata-audit | SEO / structured data | 8/9 |
Read it honestly
This ranks the latency microbenchmark (one controlled instrument) and a real-work capability suite (11 harvested tasks) as two separate claims. TodoMVC never feeds the capability number, and a 2-rep screening result should not be ordered finely against another 2-rep row.
The benchmark repo is the source of truth; this page is a dated snapshot of it. Full contracts, raw per-run JSON, and the exclusion log live at github.com/Rajeev-SG/web-automation-microbench.