# APEX-SWE ![Category](https://img.shields.io/badge/Category-Coding%20Agent-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Pass%401-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-200-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-2026-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/pdf/2601.08806) [![GitHub](https://img.shields.io/badge/GitHub-Mercor--Intelligence%2Fapex--swe-181717?logo=github&logoColor=white)](https://github.com/Mercor-Intelligence/apex-swe) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/mercor/APEX-SWE) > **Quick answer:** APEX-SWE is a software engineering agent benchmark built by Mercor in collaboration with Cognition, scoring models on Pass@1 across a 200-task heldout set (with a 50-task subset released publicly on HuggingFace). Grok 4.6 scores 56.4% as of August 2026. ## At a Glance **What it tests:** A coding agent's ability to resolve real software engineering tasks correctly on the first attempt (Pass@1), without retries or multiple sampled attempts. **Why it matters:** Pass@1 scoring is a stricter, more production-relevant signal than pass@k metrics that allow multiple sampled attempts — it more closely reflects single-shot reliability in an actual developer workflow. **Known limitations:** The full 200-task evaluation set is heldout and not public; only a 50-task development subset is released on HuggingFace, so full-set scores can't be independently reproduced. ## What APEX-SWE Measures APEX-SWE was built by Mercor, a talent and evaluation infrastructure company, in collaboration with Cognition (maker of the Devin coding agent). It evaluates coding agents on real-world software engineering tasks, scored strictly on Pass@1 — whether the agent's first and only attempt resolves the task correctly, verified against held-out test suites. A 200-task heldout evaluation set is used for official leaderboard scoring, with a smaller 50-task development subset published publicly to let teams sanity-check their own harness before requesting an official score. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Coding agent | | Metric | % Pass@1 | | Number of tasks | 200 (heldout) / 50 (public dev set) | | Saturation | Low | | Created by | Mercor (with Cognition) | | Source paper | [Mercor et al. 2026](https://arxiv.org/pdf/2601.08806) | | GitHub | [Mercor-Intelligence/apex-swe](https://github.com/Mercor-Intelligence/apex-swe) | | Dataset | [mercor/APEX-SWE on HuggingFace](https://huggingface.co/datasets/mercor/APEX-SWE) | ## How APEX-SWE Is Scored Each of the 200 heldout tasks is attempted exactly once (Pass@1); the model's score is the percentage of tasks resolved correctly and verified via automated test execution, with no retries counted toward the score. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Grok 4.6](/models/xai/grok-4-6) | 56.4% | [xAI Grok 4.6 announcement](https://x.ai/news/grok-4-6) | 2026-08 | *Score sourced from xAI's Grok 4.6 announcement, August 2026.* ## APEX-SWE on Benchgen > No Benchgen results yet — [be the first to run APEX-SWE](/benchmarks/mercor/apex-swe). ## APEX-SWE vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **APEX-SWE** | Pass@1 software engineering agent tasks | 200 | Low | | [DeepSWE](/benchmarks/datacurve/deepswe) | Real-world software engineering agent tasks | — | Low | | [APEX-Agents](/benchmarks/mercor/apex-agents) | General agentic task success | — | Low | | [SWE-Bench Verified](/benchmarks/princeton-nlp-openai/swe-bench-verified) | Verified real-world GitHub issue resolution | — | Medium | ## Run APEX-SWE on Your Model Benchgen lets you evaluate your own model and harness against APEX-SWE-style Pass@1 tasks and track single-shot reliability over time. ## Frequently Asked Questions

What is APEX-SWE?

APEX-SWE is a software engineering agent benchmark built by Mercor with Cognition, scoring models on Pass@1 across a 200-task heldout evaluation set.

What does a good score look like on APEX-SWE?

Grok 4.6 reports 56.4% as of August 2026, a strong result for a strict single-attempt (Pass@1) metric.

Who created APEX-SWE?

APEX-SWE was created by Mercor in collaboration with Cognition; see the [paper](https://arxiv.org/pdf/2601.08806) and [GitHub repo](https://github.com/Mercor-Intelligence/apex-swe).

Is the APEX-SWE dataset public?

A 50-task development subset is published on [HuggingFace](https://huggingface.co/datasets/mercor/APEX-SWE); the full 200-task evaluation set used for official scoring is heldout.

APEX-SWE — Results

APEX-SWE — Evaluations