Terminal-Bench Hard
1 phaseActive
A curated set of 89 hard, realistic command-line tasks used to stress-test AI agents' terminal skills, where frontier models score under 65%.
Overview
Terminal-Bench Hard
Quick answer: Terminal-Bench Hard is a carefully curated 89-task subset of Terminal-Bench 2.0, composed of hard, realistic command-line tasks inspired by real engineering workflows, each with a unique environment, human-written reference solution, and comprehensive tests — and one where frontier models and agents currently score below 65%.
At a Glance
What it tests: Whether AI agents can complete hard, realistic tasks entirely inside a command-line terminal environment, from environment setup and debugging to multi-step system administration.
Why it matters: Terminal use is a foundational skill for autonomous coding and DevOps agents; Terminal-Bench Hard specifically curates the most difficult tasks from the broader Terminal-Bench suite so that even frontier agents are far from saturating it.
Known limitations: As a curated "hard" subset, task selection involves subjective difficulty judgments, and the benchmark is released with tagged versions on the "Harbor Hub," meaning task composition can evolve between releases.
What Terminal-Bench Hard Measures
Terminal-Bench Hard is drawn from Terminal-Bench 2.0, described by its authors as "a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows." Every task ships with its own isolated environment, a human-written reference solution, and a comprehensive automated test suite used to grade agent transcripts.
The benchmark is designed to sit at the difficulty ceiling of current terminal-agent capability: the authors report that frontier models and agents score less than 65% on the full hard set, making it a meaningfully unsaturated stress test for CLI-based agentic workflows.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Agent / command-line task completion |
| Metric | Task success rate (%), graded by automated test suites per task |
| Number of tasks | 89 curated hard tasks |
| Saturation | Low — frontier models and agents score under 65% |
| Created by | Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, et al. (Stanford, Laude Institute) |
| Source paper | Merrill et al. 2026 |
| GitHub | harbor-framework/terminal-bench |
How Terminal-Bench Hard Is Scored
Each of the 89 tasks is graded via an automated test suite that checks the final state of the terminal environment against the task's requirements after the agent finishes acting; the headline metric is the percentage of the 89 tasks completed successfully.
State-of-the-Art Results
The Terminal-Bench 2.0 paper reports that frontier models and agents score less than 65% on the hard 89-task set, establishing it as a deliberately unsaturated benchmark. See the official site and GitHub repository for current per-agent leaderboard results.
Terminal-Bench Hard on Benchgen
No Benchgen results yet — be the first to run Terminal-Bench Hard.
Terminal-Bench Hard vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| Terminal-Bench Hard | Hard, realistic CLI agent tasks | 89 | low |
| Terminal-Bench 3.0 | Broader CLI agent task suite | — | — |
| TerminalBench 2.1 | Earlier-generation CLI agent task suite | — | — |
Terminal-Bench Hard is distinguished from the broader-scope Terminal-Bench releases by deliberately narrowing to the 89 hardest, most realistic tasks, making it a sharper signal for whether an agent is approaching genuine terminal-competence rather than passing easier warm-up tasks.
Run Terminal-Bench Hard on Your Model
Benchgen lets teams evaluate their own agent against Terminal-Bench Hard's 89 curated tasks, surfacing exactly where command-line reasoning and multi-step execution break down relative to frontier agents.