SWE Atlas – Test Writing — Results
| Rank | Model | Score |
|---|---|---|
| 1 | hy4-preview | 57.8 |
SWE Atlas – Test Writing
1 phaseActive
Scale AI's SWE Atlas sub-benchmark testing agentic test-writing quality on real repositories. 90 tasks.
Overview
SWE Atlas – Test Writing
Quick answer: SWE Atlas – Test Writing is one of three sub-benchmarks in Scale AI's SWE Atlas suite. It tests whether a coding agent can write correct, meaningful tests for existing code — 90 tasks scored with programmatic checks plus rubric-based assessment.
At a Glance
What it tests: Whether an AI coding agent can write tests that meaningfully exercise existing code — not just tests that pass trivially, but ones that actually catch regressions.
Why it matters: Writing good tests is a distinct skill from writing correct code; agents that can extend test coverage on an existing codebase are far more useful in real engineering workflows.
Known limitations: Small task count (90); rubric grading of "test quality" is inherently more subjective than binary pass/fail checks.
What SWE Atlas – Test Writing Measures
This slice of Scale AI's SWE Atlas suite drops an agent into a real repository and asks it to write tests for specified existing functionality. Grading combines programmatic checks (does the test suite run, does it pass against the correct implementation, does it fail against seeded bugs) with expert rubric review of test quality — coverage of edge cases, clarity, and whether the tests would catch a realistic regression.
A high score indicates an agent can produce tests that genuinely strengthen a codebase's safety net, not just tests that trivially pass. This complements the Codebase Q&A and Refactoring slices of the same suite.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Coding |
| Metric | Accuracy / rubric score |
| Number of tasks | 90 |
| Saturation | Low |
| Created by | Raghavendra et al. (Scale AI) |
| Source paper | Raghavendra et al. 2026 |
| GitHub | scaleapi/SWE-Atlas |
How SWE Atlas – Test Writing Is Scored
Scores combine automated test-execution checks (pass against correct code, fail against seeded bugs) with expert rubric assessment of test quality, reported as a percentage.
State-of-the-Art Results
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | Hy4 Preview | 57.8% | Tencent Hunyuan model card | 2026-08 |
Scores sourced from published technical reports and model cards. Results depend on harness, prompt format, and effort settings — see each source for methodology.
SWE Atlas – Test Writing on Benchgen
No Benchgen results yet — be the first to run SWE Atlas – Test Writing.
SWE Atlas – Test Writing vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| SWE Atlas – Test Writing | Agentic test-writing quality | 90 | Low |
| SWE Atlas – Codebase Q&A | Repository comprehension | 124 | Low |
| SWE Atlas – Refactoring | Code refactoring quality | 70 | Low |
Use SWE Atlas – Test Writing to evaluate an agent's ability to strengthen an existing codebase's test coverage, distinct from its ability to resolve issues end-to-end.
Run SWE Atlas – Test Writing on Your Model
Benchgen lets teams run SWE Atlas – Test Writing against their own model versions, compare results across runs and harnesses, and catch regressions in test-generation quality over time.
Related Models & Benchmarks
- Top performers: Hy4 Preview
- Related benchmarks: SWE Atlas – Codebase Q&A · SWE Atlas – Refactoring