Aider — Results
| Rank | Model | Score |
|---|---|---|
| 1 | deepseek-v2-5 | 0.722 |
| 2 | qwen3-235b-a22b | 0.618 |
| 3 | qwen2-5-coder-7b-instruct | 0.556 |
| 4 | qwen3-32b | 0.502 |
Aider
1 phaseActive
Paul Gauthier's code-editing benchmark: 133 Exercism Python exercises scored by unit test pass rate. Tests end-to-end AI code editing capability.
Overview
Aider
Quick answer: The Aider benchmark evaluates AI models on 133 Python coding exercises sourced from Exercism, measuring whether models can translate natural language coding requests into working code that passes automated unit tests. It tests end-to-end code editing ability — writing new code, modifying existing files, and formatting changes for automated saving. DeepSeek-V2.5 currently leads at 72.2%.
At a Glance
What it tests: End-to-end AI code editing — generating or modifying Python code so that it passes a suite of unit tests, covering the full pipeline from understanding a natural language problem statement to producing correctly formatted file changes.
Why it matters: Unlike benchmarks that evaluate isolated code generation, Aider measures a model's ability to interact with an existing codebase, apply targeted edits, and produce output that can be automatically saved — closely mirroring real-world AI-assisted development workflows.
Known limitations: The benchmark uses only Python exercises from a single source (Exercism), which limits language and domain diversity. The Aider Polyglot variant (225 exercises across C++, Go, Java, JavaScript, Python, and Rust) addresses this but is a separate evaluation.
What Aider Measures
Aider presents a model with a natural language description of a coding task drawn from Exercism's Python exercise repository. The model must generate code that passes the associated unit tests. The evaluation is end-to-end: the model's output must be correctly formatted so that it can be automatically applied to a file on disk, and the resulting code must pass all tests without human intervention.
The benchmark specifically tests whether models can edit existing code rather than just generate code from scratch. This makes it more representative of real development tasks where a partially written file exists and the model must make targeted changes. The harness handles file I/O and test execution automatically.
The Aider Polyglot variant extends this to 225 exercises spanning six programming languages and is tracked separately on the same leaderboard.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Coding |
| Metric | Pass rate (fraction of exercises passing all unit tests) |
| Number of tasks | 133 (Python); 225 (Polyglot variant) |
| Language | Python (core); C++, Go, Java, JavaScript, Python, Rust (Polyglot) |
| Source | Exercism exercise repository |
| Saturation | Medium |
| Created by | Paul Gauthier (Aider AI) |
| GitHub | Aider-AI/aider |
How Aider Is Scored
Each exercise is scored as pass or fail: the model must produce code that passes all unit tests for that exercise. The overall score is the fraction of exercises where all tests pass. No partial credit is awarded for partially passing test suites.
The evaluation is fully automated — model outputs are applied to files and tests are run without human review. A "good" score on Aider is above 60%; the top frontier models currently cluster in the 60–72% range, indicating the benchmark still meaningfully differentiates capable models.
Aider on Benchgen
No Benchgen results yet — be the first to run Aider.
Aider vs Other Code Editing Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| Aider | End-to-end code editing (Python exercises) | 133 | Medium |
| HumanEval | Function completion from docstring | 164 | High |
| BigCodeBench | Realistic function synthesis with libraries | 1140 | Medium |
| SWE-bench Verified | Real GitHub issue resolution | 500 | Low |
| LiveCodeBench | Competitive programming problems | Ongoing | Low |
Aider sits between isolated function synthesis (HumanEval) and full repository-level issue resolution (SWE-bench). It is particularly useful for evaluating models that will be used in AI-assisted coding workflows where file editing is a core requirement.
Run Aider on Your Model
Benchgen lets engineering teams run Aider against specific model versions, compare results across releases, and detect regressions before they reach production — rather than relying on vendor-reported one-time numbers. Track how your fine-tuned or deployed model performs on Aider continuously.
Related Models & Benchmarks
- Top performers: DeepSeek-V2.5 · Qwen3-235B-A22B · Qwen2.5-Coder 7B Instruct
- Related benchmarks: HumanEval · BigCodeBench · LiveCodeBench · SWE-bench Verified
Frequently Asked Questions
What is the Aider benchmark?
The Aider benchmark evaluates AI models on 133 Python coding exercises from Exercism. Models must translate natural language problem descriptions into code that passes automated unit tests, with outputs formatted for automatic file editing. It is maintained by Paul Gauthier, creator of the Aider AI coding assistant.What does a good Aider score look like?
Scores above 60% are considered strong. Top frontier models score in the 61–72% range as of mid-2026. The benchmark is not saturated — meaningful differences exist between capable models.Who created the Aider benchmark?
Paul Gauthier, creator of the Aider open-source AI coding assistant. The benchmark was developed to evaluate models for use with the Aider tool. See the GitHub repository for methodology details.Is the Aider benchmark saturated?
No. With the top model scoring 72.2% as of 2026, there is meaningful headroom. The Aider Polyglot variant (six languages) is even harder and less saturated.How does Aider differ from HumanEval?
HumanEval tests isolated function completion from a docstring. Aider tests end-to-end code editing: the model must understand a task description, produce correctly formatted file changes, and pass a full test suite — a more realistic simulation of AI-assisted development.Benchmark methodology from Aider-AI/aider. Last updated 2026-07-23.