This page holds the method, limits and per task numbers behind every comparative claim on this site: okyra against the agent-browser CLI, measured on the same tasks in the same browser.
Every raw run, the harness and the report script are public at github.com/stavthal/okyra-benchmarks. Every number below is computed from those JSON files, one per run, exactly as Claude Code wrote them.
Headline
Average of the six task medians, 5 runs per tool per task:
| Measure | okyra | agent-browser CLI | Difference |
|---|---|---|---|
| Model turns per task | 3.3 | 6.0 | 44% fewer |
| Model cost per task (USD) | $0.059 | $0.114 | $0.054 less, 48% less |
| Model API time per task | 9.3 s | 18.1 s | 8.8 s less |
| Input tokens per task | 133,786 | 253,696 | 47% fewer |
| Output tokens per task | 508 | 919 | 45% fewer |
| Correct answers | 29 of 30 | 30 of 30 |
Cost saving on every task: $0.022 to $0.080 per task, 26% to 58%. Model API time is time spent waiting on the model, not end to end task time. okyra's one miss was a run in which the test site returned an error page.1
Method
| Date | 2026-10-13, both rounds |
|---|---|
| Harness | Headless Claude Code, claude -p --output-format json, script harness/run.sh, summary harness/report.py |
| Model | claude-opus-5-5, standard speed, fast mode off |
| Cost basis | total_cost_usd as Claude Code reports it, at list price |
| okyra | The build of 2026-10-13, through its MCP server |
| agent-browser | 0.39.0, the latest release on that date, through its CLI with its bundled core skill, which its README recommends for coding agents. This was its best arm. Its MCP server also ran on T2 and T3 and cost more than its CLI, so we compare against the CLI. |
| Browser | The same Brave install for both, both tools warm (daemon and browser up before each run) |
| Runs | 5 per tool per task. Medians reported, ranges kept |
| Correctness | Each answer checked by correct() in the report script |
| Tuning | okyra was tuned after seeing T2 and T3. T4 to T7 were new to both tools |
Input tokens are input plus cache creation plus cache read tokens. Model time is duration_api_ms.
Per task
Medians of 5 runs, okyra against the agent-browser CLI arm.
| Task | okyra turns | agent-browser turns | okyra cost | agent-browser cost |
|---|---|---|---|---|
| T2 Wikipedia search, read the infobox (tuned) | 4 | 7 | $0.050 | $0.118 |
| T3 Fill and submit an order form (tuned) | 4 | 8 | $0.078 | $0.158 |
| T4 Log in | 3 | 4 | $0.060 | $0.082 |
| T5 Single page app: add, complete, count | 3 | 6 | $0.051 | $0.117 |
| T6 Pagination to page 3 | 3 | 6 | $0.069 | $0.100 |
| T7 Read late loading content | 3 | 5 | $0.046 | $0.106 |
On the four tasks okyra was not tuned for (T4 to T7), it used 43% fewer model turns and 44% lower model cost per task than agent-browser in our tests.1
Tasks: T2 Wikipedia search and read the infobox; T3 fill and submit the httpbin order form; T4 log in at the-internet.herokuapp.com; T5 TodoMVC React app, add three todos, complete one, read the counter; T6 quotes.toscrape.com, page to page 3, read the first author; T7 the-internet dynamic loading, read late content.
Cold cache
The first one or two runs of most cells cost $0.23 to $0.34 for both tools, because they wrote 25,000 to 40,000 tokens into the prompt cache; later runs read it. A cold cache costs about the same extra for either tool, so it shrinks the percentage but not the dollar saving.
| Way of averaging | okyra | agent-browser CLI | Saving |
|---|---|---|---|
| Average of the six task medians (mostly warm cache) | $0.059 | $0.114 | $0.054, 48% |
| Mean of all 30 runs (cold and warm) | $0.118 | $0.169 | $0.052, 31% |
Limits
- Small sample. 5 runs per tool per task, six tasks, one day. The spread between runs is wide.
- One model, one harness. Claude Opus 5.5 in Claude Code only. Not measured on other Claude models, GPT or Gemini models, nor in custom agent loops, LangChain, the Agent SDK or browser-use.
- Versions and dates. agent-browser 0.39.0 and okyra as built on 2026-10-13. A later agent-browser release can change the result. agent-browser has a batch command, so its result depends on how agents use it.
- Tuning. okyra was tuned after seeing T2 and T3. T4 to T7 were new to both tools and still show 25% to 57% savings.
- The okyra miss. One T4 run failed because the test site served an error page; okyra returned it and the agent reported the outage. It is counted as incorrect.
- Wall clock time is not claimed. End to end time includes harness start and shell overhead that was not isolated.
- Prompt hook. The same Claude Code prompt hook ran inside every run of both tools.
- Not compared. Playwright MCP, Chrome DevTools MCP, browser-use and Stagehand were not benchmarked. We claim no saving against them.
- Rerunning. The harness needs the okyra binary on your PATH, so a third party can rerun it once okyra is downloadable.
Freshness
We rerun the benchmark within 30 days of each new agent-browser minor or major release, or every 90 days, whichever comes first, and update this page. Until then the numbers stay labelled with agent-browser 0.39.0 and 13 October 2026. If a new release closes the gap, the claims come down.
To measure okyra on your own tasks, start with the okyra quick start, then write the steps as a plan that runs in one call.
1. Our benchmark, run 2026-10-13: six browser tasks, 5 runs per tool per task, medians reported. Claude Opus 5.5 (claude-opus-5-5) in Claude Code, costs at list price as Claude Code reports them. okyra build of 2026-10-13 through its MCP server; Vercel agent-browser 0.39.0 (latest release on that date) through its CLI with its bundled core skill. Same browser for both, both warm. okyra was tuned after seeing two of the six tasks. Method, limits, raw results and scripts: our benchmark page and github.com/stavthal/okyra-benchmarks.