SWE-Lancer
1 phaseActive
OpenAI benchmark of 1,400+ real Upwork freelance software engineering tasks worth $1M total payout, including both IC coding tasks and managerial decisions.
Overview
SWE-Lancer
Quick answer: SWE-Lancer is an OpenAI benchmark of over 1,400 real freelance software engineering tasks sourced from Upwork, collectively valued at $1 million USD in real-world payouts, testing whether frontier LLMs can both write shippable code and make sound engineering-management decisions.
At a Glance
What it tests: Whether a model can independently resolve real freelance software tasks (from $50 bug fixes to $32,000 feature builds) end-to-end, and separately, whether it can make good managerial judgment calls between competing technical proposals.
Why it matters: By mapping model performance directly to the dollar value of the original freelance payout, SWE-Lancer connects benchmark accuracy to a concrete economic interpretation — how much real money a model could plausibly earn doing this work today.
Known limitations: Tasks are drawn from one freelance platform (Upwork) and one large closed-source product codebase (Expensify), so results may not generalize evenly across all engineering domains or company-specific tooling conventions.
What SWE-Lancer Measures
SWE-Lancer splits its 1,400+ tasks into two categories. Independent Engineer (IC) tasks are real bug-fix and feature-implementation tickets, graded with end-to-end tests that were triple-verified by experienced software engineers to ensure grading fidelity matches real acceptance criteria. Managerial tasks instead present a model with multiple competing technical implementation proposals for a task and ask it to select the best one, with correctness assessed against the actual choice made by the original hired engineering manager.
Because every task carries its real Upwork payout amount, SWE-Lancer converts raw resolution rate into a dollar figure — directly answering "how much of this $1 million could a model earn?" rather than just reporting an abstract percentage. The authors found that frontier models at publication time were still unable to resolve the majority of tasks, leaving most of the $1 million unclaimed. A smaller, public evaluation split called SWE-Lancer Diamond is open-sourced for reproducible third-party evaluation.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Coding / real-world freelance software engineering |
| Metric | Resolved task value (USD) and task-level pass rate (%) |
| Number of tasks | 1,400+ (full set); SWE-Lancer Diamond is the public eval split |
| Saturation | Low — frontier models still fail to resolve most tasks |
| Created by | Samuel Miserendino, Michele Wang, Tejal Patwardhan, Johannes Heidecke (OpenAI) |
| Source paper | Miserendino et al. 2025 |
| GitHub | openai/SWELancer-Benchmark |
How SWE-Lancer Is Scored
IC tasks are graded pass/fail against triple-verified end-to-end test suites; managerial tasks are graded by matching the model's selected proposal against the real manager's historical decision. Aggregate results are reported both as a percentage pass rate and as the cumulative dollar value of tasks resolved out of the $1 million total pool, giving a direct economic framing alongside the standard accuracy metric.
State-of-the-Art Results
Scores are typically reported as a percentage of the $1M total value earned, and shift by model release. See the official GitHub repository and paper for the full per-model breakdown across IC and managerial task splits.
SWE-Lancer on Benchgen
No Benchgen results yet — be the first to run SWE-Lancer.
SWE-Lancer vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| SWE-Lancer | Real freelance IC + managerial engineering tasks | 1,400+ | low |
| SWE-bench Verified | GitHub issue-to-PR resolution | 500 | medium |
| RepoBench | Repository-level code completion | — | medium |
SWE-Lancer is distinguished from SWE-bench Verified by its real freelance-marketplace payout framing and its unique managerial decision-making task type, going beyond pure code-patch resolution.
Run SWE-Lancer on Your Model
Benchgen lets teams evaluate their own coding model or agent on SWE-Lancer-style IC and managerial tasks, tracking both pass rate and dollar-value-earned trends across releases.