StaminaBench
1 phaseActive
Black-box benchmark for multi-turn coding-agent endurance — consecutive change requests survived before failure.
Overview
StaminaBench
Quick answer: StaminaBench is an Amazon Science benchmark that measures coding-agent "stamina" — how many consecutive change-request turns (up to 100) an agent can sustain before it fails a request — as a black-box test of long-horizon reliability rather than single-shot task accuracy. It finds that access to test feedback and retry capability improves the number of turns survived by up to 12×, and that even the strongest models show a 6× gap between their best and worst harness.
At a Glance
What it tests: How many consecutive, iterative change-request turns a coding agent can handle in a single session before it breaks down — a proxy for long-horizon reliability rather than one-shot task accuracy.
Why it matters: Real coding-agent usage is rarely single-shot; developers issue a stream of follow-up change requests in one session. StaminaBench directly measures the failure mode that matters in that setting — gradual degradation and eventual breakdown — rather than pass/fail on isolated tasks.
Known limitations: As a black-box benchmark, it doesn't diagnose why an agent fails at turn N — only that it does. Detailed per-model, per-harness turn counts are reported in the source material but not fully reproduced here.
What StaminaBench Measures
StaminaBench places a coding agent into a long, iterative session and issues up to 100 sequential change requests, each building on the previous state of the code. The benchmark tracks the number of consecutive turns the agent can complete successfully before it fails a request — whether by introducing a regression, losing track of prior context, or simply refusing/erroring out. This "stamina" framing directly targets a failure mode that static, single-shot benchmarks can't see: agents that perform well on a first request often degrade steadily as session length grows, due to context management issues, compounding errors, or harness-level state drift.
The benchmark's headline findings are both about harness design rather than raw model capability: giving the agent access to test feedback and retry capability (i.e., letting it see whether its own change broke something and try again) improves the number of turns survived by up to 12×. Separately, even the strongest models show as much as a 6× gap in turns survived between their best-performing harness and their worst — reinforcing that endurance under iterative use is substantially a harness property, not purely a model property.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Agent (coding, long-horizon endurance) |
| Metric | Consecutive turns survived (count, capped at 100) |
| Evaluation style | Black-box, iterative change-request sequence |
| Saturation | Low |
| Created by | Amazon Science |
| Source | Amazon Science publication, 2026 |
| GitHub | Not public |
| Dataset | Not public |
How StaminaBench Is Scored
Each run starts a coding-agent session and issues change requests one at a time, in sequence, up to a maximum of 100 turns. The agent "survives" a turn if it completes the requested change without breaking existing functionality or losing coherent state; the run ends the moment the agent fails a turn (or reaches the 100-turn cap). The reported score is simply the number of turns survived — higher is better, and the 100-turn cap represents a ceiling rather than a typical result for most configurations.
Because the benchmark isolates test feedback/retry access and harness choice as the two biggest levers on turns survived (12× and 6× respectively), it's best read as a harness-design diagnostic: teams building long-session coding agents can use it to check whether their retry/verification loop is actually extending session stamina.
StaminaBench on Benchgen
No Benchgen results yet — be the first to run StaminaBench.
StaminaBench vs Other Benchmarks
| Benchmark | What it tests | Saturation |
|---|---|---|
| StaminaBench | Consecutive change-request turns survived (endurance) | Low |
| SWE-Marathon | Long-horizon software engineering task completion | Low |
| SWE-bench Pro | Harder single-shot software engineering tasks | Low |
| Harness-Bench | Harness quality on a fixed academic task suite | Low |
StaminaBench and SWE-Marathon both target long-horizon coding-agent behavior, but StaminaBench specifically isolates iterative, multi-turn endurance (how many follow-up requests an agent can survive) rather than completion of a single extended task. Use StaminaBench when evaluating whether your agent's retry and verification loop holds up across a realistic multi-turn development session.
Run StaminaBench on Your Model
Benchgen lets teams run their own coding agent through an iterative, multi-turn change-request sequence and track how many turns it survives across model or harness updates — catching stamina regressions (e.g. a retry-loop change that quietly shortens session length) before they reach production developers.
Related Models & Benchmarks
- Related benchmarks: SWE-Marathon · SWE-bench Pro · Harness-Bench