# SWE Atlas – Test Writing ![Category](https://img.shields.io/badge/Category-Coding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy%2FRubric-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-90-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-2026-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2605.08366) [![GitHub](https://img.shields.io/badge/GitHub-scaleapi%2FSWE--Atlas-181717?logo=github&logoColor=white)](https://github.com/scaleapi/SWE-Atlas) > **Quick answer:** SWE Atlas – Test Writing is one of three sub-benchmarks in Scale AI's SWE Atlas suite. It tests whether a coding agent can write correct, meaningful tests for existing code — 90 tasks scored with programmatic checks plus rubric-based assessment. ## At a Glance **What it tests:** Whether an AI coding agent can write tests that meaningfully exercise existing code — not just tests that pass trivially, but ones that actually catch regressions. **Why it matters:** Writing good tests is a distinct skill from writing correct code; agents that can extend test coverage on an existing codebase are far more useful in real engineering workflows. **Known limitations:** Small task count (90); rubric grading of "test quality" is inherently more subjective than binary pass/fail checks. ## What SWE Atlas – Test Writing Measures This slice of Scale AI's SWE Atlas suite drops an agent into a real repository and asks it to write tests for specified existing functionality. Grading combines programmatic checks (does the test suite run, does it pass against the correct implementation, does it fail against seeded bugs) with expert rubric review of test quality — coverage of edge cases, clarity, and whether the tests would catch a realistic regression. A high score indicates an agent can produce tests that genuinely strengthen a codebase's safety net, not just tests that trivially pass. This complements the Codebase Q&A and Refactoring slices of the same suite. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Coding | | Metric | Accuracy / rubric score | | Number of tasks | 90 | | Saturation | Low | | Created by | Raghavendra et al. (Scale AI) | | Source paper | [Raghavendra et al. 2026](https://arxiv.org/abs/2605.08366) | | GitHub | [scaleapi/SWE-Atlas](https://github.com/scaleapi/SWE-Atlas) | ## How SWE Atlas – Test Writing Is Scored Scores combine automated test-execution checks (pass against correct code, fail against seeded bugs) with expert rubric assessment of test quality, reported as a percentage. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Hy4 Preview](/models/hungyuan/hy4-preview) | 57.8% | [Tencent Hunyuan model card](https://huggingface.co/tencent/Hy4-preview) | 2026-08 | *Scores sourced from published technical reports and model cards. Results depend on harness, prompt format, and effort settings — see each source for methodology.* ## SWE Atlas – Test Writing on Benchgen > No Benchgen results yet — [be the first to run SWE Atlas – Test Writing](/benchmarks/raghavendra-et-al-scale-ai/swe-atlas-test-writing). ## SWE Atlas – Test Writing vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **SWE Atlas – Test Writing** | Agentic test-writing quality | 90 | Low | | [SWE Atlas – Codebase Q&A](/benchmarks/raghavendra-et-al-scale-ai/swe-atlas-codebase-qa) | Repository comprehension | 124 | Low | | [SWE Atlas – Refactoring](/benchmarks/raghavendra-et-al-scale-ai/swe-atlas-refactoring) | Code refactoring quality | 70 | Low | Use SWE Atlas – Test Writing to evaluate an agent's ability to strengthen an existing codebase's test coverage, distinct from its ability to resolve issues end-to-end. ## Run SWE Atlas – Test Writing on Your Model Benchgen lets teams run SWE Atlas – Test Writing against their own model versions, compare results across runs and harnesses, and catch regressions in test-generation quality over time. ## Related Models & Benchmarks - Top performers: [Hy4 Preview](/models/hungyuan/hy4-preview) - Related benchmarks: [SWE Atlas – Codebase Q&A](/benchmarks/raghavendra-et-al-scale-ai/swe-atlas-codebase-qa) · [SWE Atlas – Refactoring](/benchmarks/raghavendra-et-al-scale-ai/swe-atlas-refactoring) ## Frequently Asked Questions

What is SWE Atlas – Test Writing?

It's a sub-benchmark of Scale AI's SWE Atlas suite that tests whether a coding agent can write correct, meaningful tests for existing code, across 90 tasks.

What does a good SWE Atlas – Test Writing score look like?

As of August 2026, frontier models score in the 45–58% range; scores above 55% represent strong test-writing capability.

Who created SWE Atlas?

SWE Atlas was created by Scale AI (Raghavendra et al.), published in 2026.

SWE Atlas – Test Writing — Results

SWE Atlas – Test Writing — Phases

SWE Atlas – Test Writing