DeepSWE — Results
| Rank | Model | Score |
|---|---|---|
| 1 | deepseek-v4-1-flash | 74.2 |
| 2 | deepseek-v4-1-flash | 74.2 |
| 3 | gpt-6-astra | 74.1 |
| 4 | grok-4-7 | 71 |
| 5 | gpt-6-sol | 68.8 |
| 6 | kimi-k3 | 67.5 |
| 7 | glm-5-3 | 66.9 |
| 8 | gpt-6-luna | 66.6 |
| 9 | grok-4-6 | 65.9 |
| 10 | nex-n2-5-max | 65.6 |
| 11 | hy4-preview | 64.3 |
| 12 | glm-5-3-flash | 63.4 |
| 13 | deepseek-v4-pro | 62.7 |
| 14 | muse-spark-1-2 | 59.3 |
| 15 | deepseek-v4-flash-vision-exp | 59.3 |
| 16 | qwen3-8-flash-next | 58.7 |
| 17 | qwen3-8-max | 56.6 |
| 18 | ornith-1-5-397b | 56 |
| 19 | nex-n2-5-pro | 55.8 |
| 20 | qwen3-8-27b | 42.2 |
| 21 | ornith-1-5-35b-a3b | 22 |
DeepSWE
1 phaseActive
Real-world software engineering agent benchmark curated by DataCurve, testing end-to-end coding-agent workflows. Metric: % task success.
Overview
DeepSWE
Quick answer: DeepSWE is a real-world software engineering agent benchmark curated by DataCurve, testing end-to-end coding agent workflows against verified ground-truth solutions. It's positioned alongside benchmarks like SWE-Bench as a harder, more diverse test of agentic coding capability. Kimi K3 scores 67.5% as of July 2026.
At a Glance
What it tests: An AI coding agent's ability to resolve real-world software engineering tasks end-to-end — understanding a codebase, implementing a fix or feature, and verifying correctness.
Why it matters: As SWE-Bench-style benchmarks approach saturation for frontier models, DeepSWE provides an independently curated task pool from DataCurve to cross-validate agentic coding performance.
Known limitations: As a newer, vendor-curated benchmark, public documentation of methodology and task composition is limited compared to established academic benchmarks like SWE-Bench.
What DeepSWE Measures
DeepSWE evaluates coding agents on real-world software engineering tasks curated by DataCurve, a data-curation company focused on agentic and coding evaluation datasets. Tasks are designed to mirror the type of work a software engineer performs day-to-day — bug fixes, feature implementation, and refactors — verified against reference solutions or test suites.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Coding agent |
| Metric | % task success |
| Saturation | Low |
| Created by | DataCurve |
| Dataset | datacurve/deep-swe on HuggingFace |
| Project | deepswe.datacurve.ai |
How DeepSWE Is Scored
Agents attempt each software engineering task and are scored on % of tasks resolved correctly, typically verified via automated test execution against the target repository.
State-of-the-Art Results
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | Kimi K3 | 67.5% | Kimi K3 technical report | 2026-07 |
Score sourced from Moonshot AI's Kimi K3 announcement, July 2026.
DeepSWE on Benchgen
No Benchgen results yet — be the first to run DeepSWE.
DeepSWE vs Other Benchmarks
| Benchmark | What it tests | Saturation |
|---|---|---|
| DeepSWE | Real-world software engineering agent tasks | Low |
| SWE-Bench Pro | Professional-grade SWE agent tasks | Low |
| SWE-Bench Verified | Verified real-world GitHub issue resolution | Medium |
| SWE-Marathon | Long-horizon software engineering | Low |
Run DeepSWE on Your Model
Benchgen lets you run DeepSWE against your own coding agent, tracking task success rate over time to catch regressions from model or harness updates.