SWE Atlas – Refactoring — Results
| Rank | Model | Score |
|---|---|---|
| 1 | hy4-preview | 53.3 |
SWE Atlas – Refactoring
1 phaseActive
Scale AI's SWE Atlas sub-benchmark testing agentic code-refactoring quality on real repositories. 70 tasks.
Overview
SWE Atlas – Refactoring
Quick answer: SWE Atlas – Refactoring is one of three sub-benchmarks in Scale AI's SWE Atlas suite. It tests whether a coding agent can perform correct, high-quality code refactors — 70 tasks scored with programmatic checks plus rubric-based assessment.
At a Glance
What it tests: Whether an AI coding agent can restructure existing code (improving readability, structure, or maintainability) without changing its observable behavior.
Why it matters: Refactoring is one of the most common, highest-value real engineering tasks — and one of the easiest for an agent to get subtly wrong by silently changing behavior.
Known limitations: Small task count (70); "refactoring quality" beyond behavior-preservation is judged by rubric, which is inherently more subjective than a binary pass/fail check.
What SWE Atlas – Refactoring Measures
This slice of Scale AI's SWE Atlas suite asks an agent to refactor specified code in a real repository — for example, extracting a function, simplifying control flow, or improving naming — while preserving exact behavior. Grading combines programmatic checks (do existing tests still pass, is behavior preserved) with expert rubric review of refactoring quality (did the change actually improve the code).
A high score indicates an agent can make codebases measurably better without breaking anything — a core, everyday agentic-coding capability distinct from either comprehension (Codebase Q&A) or new-test authorship (Test Writing).
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Coding |
| Metric | Accuracy / rubric score |
| Number of tasks | 70 |
| Saturation | Low |
| Created by | Raghavendra et al. (Scale AI) |
| Source paper | Raghavendra et al. 2026 |
| GitHub | scaleapi/SWE-Atlas |
How SWE Atlas – Refactoring Is Scored
Scores combine automated behavior-preservation checks (existing tests must still pass) with expert rubric assessment of refactoring quality, reported as a percentage.
State-of-the-Art Results
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | Hy4 Preview | 53.3% | Tencent Hunyuan model card | 2026-08 |
Scores sourced from published technical reports and model cards. Results depend on harness, prompt format, and effort settings — see each source for methodology.
SWE Atlas – Refactoring on Benchgen
No Benchgen results yet — be the first to run SWE Atlas – Refactoring.
SWE Atlas – Refactoring vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| SWE Atlas – Refactoring | Code refactoring quality | 70 | Low |
| SWE Atlas – Codebase Q&A | Repository comprehension | 124 | Low |
| SWE Atlas – Test Writing | Writing tests for existing code | 90 | Low |
Use SWE Atlas – Refactoring to evaluate an agent's ability to safely improve existing code, distinct from its ability to resolve new issues from scratch.
Run SWE Atlas – Refactoring on Your Model
Benchgen lets teams run SWE Atlas – Refactoring against their own model versions, compare results across runs and harnesses, and catch regressions in refactoring quality over time.
Related Models & Benchmarks
- Top performers: Hy4 Preview
- Related benchmarks: SWE Atlas – Codebase Q&A · SWE Atlas – Test Writing