APEX-SWE — Results
| Rank | Model | Score |
|---|---|---|
| 1 | grok-4-6 | 56.4 |
APEX-SWE
1 phaseActive
Mercor and Cognition's Pass@1 software engineering agent benchmark built from a 200-task heldout set.
Overview
APEX-SWE
Quick answer: APEX-SWE is a software engineering agent benchmark built by Mercor in collaboration with Cognition, scoring models on Pass@1 across a 200-task heldout set (with a 50-task subset released publicly on HuggingFace). Grok 4.6 scores 56.4% as of August 2026.
At a Glance
What it tests: A coding agent's ability to resolve real software engineering tasks correctly on the first attempt (Pass@1), without retries or multiple sampled attempts.
Why it matters: Pass@1 scoring is a stricter, more production-relevant signal than pass@k metrics that allow multiple sampled attempts — it more closely reflects single-shot reliability in an actual developer workflow.
Known limitations: The full 200-task evaluation set is heldout and not public; only a 50-task development subset is released on HuggingFace, so full-set scores can't be independently reproduced.
What APEX-SWE Measures
APEX-SWE was built by Mercor, a talent and evaluation infrastructure company, in collaboration with Cognition (maker of the Devin coding agent). It evaluates coding agents on real-world software engineering tasks, scored strictly on Pass@1 — whether the agent's first and only attempt resolves the task correctly, verified against held-out test suites.
A 200-task heldout evaluation set is used for official leaderboard scoring, with a smaller 50-task development subset published publicly to let teams sanity-check their own harness before requesting an official score.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Coding agent |
| Metric | % Pass@1 |
| Number of tasks | 200 (heldout) / 50 (public dev set) |
| Saturation | Low |
| Created by | Mercor (with Cognition) |
| Source paper | Mercor et al. 2026 |
| GitHub | Mercor-Intelligence/apex-swe |
| Dataset | mercor/APEX-SWE on HuggingFace |
How APEX-SWE Is Scored
Each of the 200 heldout tasks is attempted exactly once (Pass@1); the model's score is the percentage of tasks resolved correctly and verified via automated test execution, with no retries counted toward the score.
State-of-the-Art Results
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | Grok 4.6 | 56.4% | xAI Grok 4.6 announcement | 2026-08 |
Score sourced from xAI's Grok 4.6 announcement, August 2026.
APEX-SWE on Benchgen
No Benchgen results yet — be the first to run APEX-SWE.
APEX-SWE vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| APEX-SWE | Pass@1 software engineering agent tasks | 200 | Low |
| DeepSWE | Real-world software engineering agent tasks | — | Low |
| APEX-Agents | General agentic task success | — | Low |
| SWE-Bench Verified | Verified real-world GitHub issue resolution | — | Medium |
Run APEX-SWE on Your Model
Benchgen lets you evaluate your own model and harness against APEX-SWE-style Pass@1 tasks and track single-shot reliability over time.