# SWE Atlas – Codebase Q&A ![Category](https://img.shields.io/badge/Category-Coding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy%2FRubric-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-124-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-2026-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2605.08366) [![GitHub](https://img.shields.io/badge/GitHub-scaleapi%2FSWE--Atlas-181717?logo=github&logoColor=white)](https://github.com/scaleapi/SWE-Atlas) > **Quick answer:** SWE Atlas – Codebase Q&A is one of three sub-benchmarks in Scale AI's SWE Atlas suite. It tests whether a coding agent can correctly answer questions about an unfamiliar codebase — 124 tasks scored with programmatic checks plus rubric-based assessment. ## At a Glance **What it tests:** Whether an AI coding agent can navigate and correctly reason about a repository it has never seen before, answering specific questions about its structure, logic, or behavior. **Why it matters:** Real engineering work usually starts with understanding an unfamiliar codebase before making any change — this measures that comprehension step in isolation from code generation. **Known limitations:** Small task count (124) relative to broader SWE benchmarks; results depend heavily on repository size/complexity chosen for each question. ## What SWE Atlas – Codebase Q&A Measures SWE Atlas is Scale AI's benchmark suite for professional coding-agent work that goes beyond simple issue resolution — it covers understanding a repository, writing tests, and performing refactors, evaluated with a mix of programmatic checks and expert rubric grading for engineering quality (not just "does it run"). The Codebase Q&A slice specifically isolates comprehension: an agent is dropped into a real repository and asked to answer a specific factual or behavioral question about it, without being asked to change any code. A high score here indicates an agent can efficiently locate relevant code, trace logic across files, and produce a correct, well-supported answer — a prerequisite skill for any longer-horizon coding task. It's a comprehension-only slice, so it doesn't test code generation or debugging on its own; pair it with the Test Writing and Refactoring slices of the same suite for a fuller picture of agentic coding capability. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Coding | | Metric | Accuracy / rubric score | | Number of tasks | 124 | | Saturation | Low | | Created by | Raghavendra et al. (Scale AI) | | Source paper | [Raghavendra et al. 2026](https://arxiv.org/abs/2605.08366) | | GitHub | [scaleapi/SWE-Atlas](https://github.com/scaleapi/SWE-Atlas) | ## How SWE Atlas – Codebase Q&A Is Scored Each task is graded with a mix of programmatic verification (where the question has a checkable factual answer) and expert rubric assessment (where the question requires nuanced explanation). Scores are reported as a percentage of tasks judged correct. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Hy4 Preview](/models/hungyuan/hy4-preview) | 64.0% | [Tencent Hunyuan model card](https://huggingface.co/tencent/Hy4-preview) | 2026-08 | *Scores sourced from published technical reports and model cards. Results depend on harness, prompt format, and effort settings — see each source for methodology.* ## SWE Atlas – Codebase Q&A on Benchgen > No Benchgen results yet — [be the first to run SWE Atlas – Codebase Q&A](/benchmarks/raghavendra-et-al-scale-ai/swe-atlas-codebase-qa). ## SWE Atlas – Codebase Q&A vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **SWE Atlas – Codebase Q&A** | Repository comprehension | 124 | Low | | [SWE Atlas – Test Writing](/benchmarks/raghavendra-et-al-scale-ai/swe-atlas-test-writing) | Writing tests for existing code | 90 | Low | | [SWE Atlas – Refactoring](/benchmarks/raghavendra-et-al-scale-ai/swe-atlas-refactoring) | Code refactoring quality | 70 | Low | | [SWE-bench Verified](/benchmarks/princeton-nlp-openai/swe-bench-verified) | End-to-end GitHub issue resolution | 500 | Medium | Use SWE Atlas – Codebase Q&A when you specifically want to isolate an agent's repository-comprehension ability from its code-editing ability — SWE-bench-style benchmarks conflate both into a single pass/fail outcome. ## Run SWE Atlas – Codebase Q&A on Your Model Benchgen lets teams run SWE Atlas – Codebase Q&A against their own model versions, compare results across runs and harnesses, and catch regressions in codebase comprehension over time — rather than relying on a single vendor-reported number. ## Related Models & Benchmarks - Top performers: [Hy4 Preview](/models/hungyuan/hy4-preview) - Related benchmarks: [SWE Atlas – Test Writing](/benchmarks/raghavendra-et-al-scale-ai/swe-atlas-test-writing) · [SWE Atlas – Refactoring](/benchmarks/raghavendra-et-al-scale-ai/swe-atlas-refactoring) ## Frequently Asked Questions

What is SWE Atlas – Codebase Q&A?

It's a sub-benchmark of Scale AI's SWE Atlas suite that tests whether a coding agent can correctly answer questions about an unfamiliar codebase, across 124 tasks.

What does a good SWE Atlas – Codebase Q&A score look like?

As of August 2026, frontier models score in the 55–65% range; scores above 60% represent strong, reliable codebase comprehension.

Who created SWE Atlas?

SWE Atlas was created by Scale AI (Raghavendra et al.), published in 2026.

SWE Atlas – Codebase Q&A — Results

SWE Atlas – Codebase Q&A — Evaluations