SWE Atlas – Codebase Q&A — Results
| Rank | Model | Score |
|---|---|---|
| 1 | hy4-preview | 64 |
SWE Atlas – Codebase Q&A
1 phaseActive
Scale AI's SWE Atlas sub-benchmark testing codebase comprehension — can an agent correctly answer questions about an unfamiliar repository? 124 tasks.
Overview
SWE Atlas – Codebase Q&A
Quick answer: SWE Atlas – Codebase Q&A is one of three sub-benchmarks in Scale AI's SWE Atlas suite. It tests whether a coding agent can correctly answer questions about an unfamiliar codebase — 124 tasks scored with programmatic checks plus rubric-based assessment.
At a Glance
What it tests: Whether an AI coding agent can navigate and correctly reason about a repository it has never seen before, answering specific questions about its structure, logic, or behavior.
Why it matters: Real engineering work usually starts with understanding an unfamiliar codebase before making any change — this measures that comprehension step in isolation from code generation.
Known limitations: Small task count (124) relative to broader SWE benchmarks; results depend heavily on repository size/complexity chosen for each question.
What SWE Atlas – Codebase Q&A Measures
SWE Atlas is Scale AI's benchmark suite for professional coding-agent work that goes beyond simple issue resolution — it covers understanding a repository, writing tests, and performing refactors, evaluated with a mix of programmatic checks and expert rubric grading for engineering quality (not just "does it run"). The Codebase Q&A slice specifically isolates comprehension: an agent is dropped into a real repository and asked to answer a specific factual or behavioral question about it, without being asked to change any code.
A high score here indicates an agent can efficiently locate relevant code, trace logic across files, and produce a correct, well-supported answer — a prerequisite skill for any longer-horizon coding task. It's a comprehension-only slice, so it doesn't test code generation or debugging on its own; pair it with the Test Writing and Refactoring slices of the same suite for a fuller picture of agentic coding capability.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Coding |
| Metric | Accuracy / rubric score |
| Number of tasks | 124 |
| Saturation | Low |
| Created by | Raghavendra et al. (Scale AI) |
| Source paper | Raghavendra et al. 2026 |
| GitHub | scaleapi/SWE-Atlas |
How SWE Atlas – Codebase Q&A Is Scored
Each task is graded with a mix of programmatic verification (where the question has a checkable factual answer) and expert rubric assessment (where the question requires nuanced explanation). Scores are reported as a percentage of tasks judged correct.
State-of-the-Art Results
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | Hy4 Preview | 64.0% | Tencent Hunyuan model card | 2026-08 |
Scores sourced from published technical reports and model cards. Results depend on harness, prompt format, and effort settings — see each source for methodology.
SWE Atlas – Codebase Q&A on Benchgen
No Benchgen results yet — be the first to run SWE Atlas – Codebase Q&A.
SWE Atlas – Codebase Q&A vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| SWE Atlas – Codebase Q&A | Repository comprehension | 124 | Low |
| SWE Atlas – Test Writing | Writing tests for existing code | 90 | Low |
| SWE Atlas – Refactoring | Code refactoring quality | 70 | Low |
| SWE-bench Verified | End-to-end GitHub issue resolution | 500 | Medium |
Use SWE Atlas – Codebase Q&A when you specifically want to isolate an agent's repository-comprehension ability from its code-editing ability — SWE-bench-style benchmarks conflate both into a single pass/fail outcome.
Run SWE Atlas – Codebase Q&A on Your Model
Benchgen lets teams run SWE Atlas – Codebase Q&A against their own model versions, compare results across runs and harnesses, and catch regressions in codebase comprehension over time — rather than relying on a single vendor-reported number.
Related Models & Benchmarks
- Top performers: Hy4 Preview
- Related benchmarks: SWE Atlas – Test Writing · SWE Atlas – Refactoring