Artificial Analysis Coding Agent Index
1 phaseActive
Public composite leaderboard comparing coding-agent harnesses on identical models — quality, cost, and time per task.
Overview
Artificial Analysis Coding Agent Index
Quick answer: The Artificial Analysis Coding Agent Index is a public, ongoing leaderboard that composites three benchmarks — DeepSWE, Terminal-Bench v2, and SWE-Atlas-QnA — to rank coding-agent harness–model combinations rather than models alone. Its most-cited finding: the same model (e.g. Claude Opus 4.7) run through different harnesses (Claude Code vs. Cursor CLI vs. OpenCode) shows a 32× swing in cost per task ($0.07–$2.26) at near-identical code quality, making harness choice one of the largest cost levers in production coding-agent deployments.
At a Glance
What it tests: Coding-agent quality and operating cost, reported per harness–model combination, holding the model constant to isolate what the harness itself changes.
Why it matters: It's one of the most visible public leaderboards explicitly built around the "same model, different harness" comparison — turning harness choice into a measurable, board-worthy line item (cost per task, time per task) rather than an invisible implementation detail.
Known limitations: As a composite index, its score aggregates three underlying benchmarks with different task distributions and difficulty profiles; teams should check the individual component scores (DeepSWE, Terminal-Bench v2, SWE-Atlas-QnA) if they care about a specific coding-agent capability rather than the blended index.
What the Coding Agent Index Measures
The Coding Agent Index blends results from three existing coding-agent benchmarks — DeepSWE, Terminal-Bench v2, and SWE-Atlas-QnA — into a single composite score, then reports that score per harness–model combination instead of per model alone. This design choice reflects a growing consensus (also seen in Harness-Bench and EnterpriseClawBench) that a model's real-world coding-agent quality is inseparable from the harness wrapped around it: tool routing, retry logic, context management, and prompt scaffolding all materially change outcomes even when the underlying model is identical.
Alongside the composite quality score, Artificial Analysis reports cost per task and time per task for each harness–model pair — because in production deployments, a harness that's marginally better on quality but dramatically more expensive or slower is often the wrong choice. The index's most widely cited data point is a 32× cost swing ($0.07 to $2.26 per task) observed across different harnesses running the same model, with near-identical resulting code quality — meaning the harness alone, not the model, was responsible for most of the cost difference.
Because it's a living index rather than a static paper, rankings shift as harnesses (Claude Code, Cursor CLI, OpenCode, and others) release updates and as new frontier models are added to the comparison grid.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Agent (coding, harness comparison) |
| Metric | Composite index score + cost ($) and time per task |
| Component benchmarks | DeepSWE, Terminal-Bench v2, SWE-Atlas-QnA |
| Leaderboard unit | Harness–model combination |
| Saturation | Low |
| Created by | Artificial Analysis |
| Leaderboard | artificialanalysis.ai/agents/coding-agents |
| Paper | Not applicable — ongoing public leaderboard, not a static paper |
| Dataset | Not public (aggregates existing benchmark datasets) |
How the Coding Agent Index Is Scored
Each harness–model combination is run against the three component benchmarks, and the resulting scores are blended into a single composite index value. Cost per task and time per task are tracked separately and reported alongside the quality score rather than folded into it, so teams can see the full quality/cost/speed trade-off for each harness–model pair rather than a single number that obscures cost differences.
The headline 32× cost gap ($0.07–$2.26 per task) was observed between harnesses running the identical model (Claude Opus 4.7) with comparable output quality — illustrating that, on this index, harness choice can be a larger lever on operating cost than model choice.
State-of-the-Art Results
| Metric | Observation | Source | Date |
|---|---|---|---|
| Cost range across harnesses (same model, Claude Opus 4.7) | $0.07 – $2.26 per task (32×) | Coding Agent Index coverage | 2026 |
| Harnesses compared | Claude Code, Cursor CLI, OpenCode | Artificial Analysis | 2026 |
The full ranked leaderboard (harness–model composite scores) updates continuously on Artificial Analysis and is not reproduced in static form here — see the live leaderboard for current standings.
Coding Agent Index on Benchgen
No Benchgen results yet — be the first to run a harness-cost comparison.
Coding Agent Index vs Other Benchmarks
| Benchmark | What it tests | Saturation |
|---|---|---|
| AA Coding Agent Index | Coding-agent quality + cost, per harness–model pair | Low |
| Harness-Bench | Harness quality on a fixed academic task suite | Low |
| EnterpriseClawBench | Enterprise agent workflows, harness–model pairs | Low |
| SWE-bench Pro | Harder software engineering tasks, model-only | Low |
The Coding Agent Index is the most visible commercial leaderboard doing harness-vs-model breakdowns, while Harness-Bench and EnterpriseClawBench are academic benchmarks with similar goals. Use the Coding Agent Index when cost-per-task and time-per-task matter as much as raw quality; use Harness-Bench when you need a controlled, oracle-checkable task suite.
Run a Harness Cost Comparison on Your Model
Benchgen lets teams benchmark their own harness–model combinations for quality, cost, and time per task — so you can make the same trade-off decisions the Coding Agent Index highlights, but against your own production configuration instead of a public reference model.
Related Models & Benchmarks
- Related models: Claude Opus 4.7 · GPT-5.5
- Related benchmarks: Harness-Bench · EnterpriseClawBench · SWE-bench Pro