Methodology
Every agent gets the same prompts, a fresh environment, and three runs per task. Details below are the plan; they may be refined before the first published results, and changes will be logged.
Tracks
Coding agents and general/computer-use agents are tested on different tasks so comparisons are fair. Where feasible we may also run T3–T5 on coding agents and report them separately.
The five tasks
| # | Track | Task |
|---|---|---|
| T1 | Coding | Fix a seeded bug in a small public repo; tests must pass. |
| T2 | Coding | Add a small feature from a 5-line ticket and open a PR. |
| T3 | General | Research brief: compare 5 products with cited, checkable sources. |
| T4 | General | Build and publish a simple one-page website from a spec. |
| T5 | General | Turn a CSV into a weekly summary document with 3 charts. |
Scoring rubric (100 points)
| Category | Points | How it is scored |
|---|---|---|
| Completion | 40 | Automated check or checklist. |
| Quality / accuracy | 25 | Blind-scored against a checklist. Each hallucinated citation is −5. |
| Autonomy | 20 | 0 interventions = 20, 1 = 14, 2 = 8, 3 or more = 0. An intervention is any human message or click beyond the initial prompt, including approvals and unsticking. |
| Cost efficiency | 10 | Relative to the median cost for that task. |
| Speed | 5 | Wall-clock time per task. |
Recorded per run
Finished (yes/no), correctness, wall time, cost, and number of human interventions.
Headline stats
Finish rate (share of 3 runs fully passing), median cost per run, average interventions, and score.
Transparency
We intend to publish prompts, dates, agent versions, and raw logs. Sponsors never alter scores, and sponsorships are always disclosed.