# SWE-Lancer ![Category](https://img.shields.io/badge/Category-Coding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-USD%20Earned-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-1%2C400%2B-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Feb%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2502.12115) [![GitHub](https://img.shields.io/badge/GitHub-openai%2FSWELancer--Benchmark-181717?logo=github&logoColor=white)](https://github.com/openai/SWELancer-Benchmark) > **Quick answer:** SWE-Lancer is an OpenAI benchmark of over 1,400 real freelance software engineering tasks sourced from Upwork, collectively valued at $1 million USD in real-world payouts, testing whether frontier LLMs can both write shippable code and make sound engineering-management decisions. ## At a Glance **What it tests:** Whether a model can independently resolve real freelance software tasks (from $50 bug fixes to $32,000 feature builds) end-to-end, and separately, whether it can make good managerial judgment calls between competing technical proposals. **Why it matters:** By mapping model performance directly to the dollar value of the original freelance payout, SWE-Lancer connects benchmark accuracy to a concrete economic interpretation — how much real money a model could plausibly earn doing this work today. **Known limitations:** Tasks are drawn from one freelance platform (Upwork) and one large closed-source product codebase (Expensify), so results may not generalize evenly across all engineering domains or company-specific tooling conventions. ## What SWE-Lancer Measures SWE-Lancer splits its 1,400+ tasks into two categories. **Independent Engineer (IC) tasks** are real bug-fix and feature-implementation tickets, graded with end-to-end tests that were triple-verified by experienced software engineers to ensure grading fidelity matches real acceptance criteria. **Managerial tasks** instead present a model with multiple competing technical implementation proposals for a task and ask it to select the best one, with correctness assessed against the actual choice made by the original hired engineering manager. Because every task carries its real Upwork payout amount, SWE-Lancer converts raw resolution rate into a dollar figure — directly answering "how much of this $1 million could a model earn?" rather than just reporting an abstract percentage. The authors found that frontier models at publication time were still unable to resolve the majority of tasks, leaving most of the $1 million unclaimed. A smaller, public evaluation split called **SWE-Lancer Diamond** is open-sourced for reproducible third-party evaluation. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Coding / real-world freelance software engineering | | Metric | Resolved task value (USD) and task-level pass rate (%) | | Number of tasks | 1,400+ (full set); SWE-Lancer Diamond is the public eval split | | Saturation | Low — frontier models still fail to resolve most tasks | | Created by | Samuel Miserendino, Michele Wang, Tejal Patwardhan, Johannes Heidecke (OpenAI) | | Source paper | [Miserendino et al. 2025](https://arxiv.org/abs/2502.12115) | | GitHub | [openai/SWELancer-Benchmark](https://github.com/openai/SWELancer-Benchmark) | ## How SWE-Lancer Is Scored IC tasks are graded pass/fail against triple-verified end-to-end test suites; managerial tasks are graded by matching the model's selected proposal against the real manager's historical decision. Aggregate results are reported both as a percentage pass rate and as the cumulative dollar value of tasks resolved out of the $1 million total pool, giving a direct economic framing alongside the standard accuracy metric. ## State-of-the-Art Results > Scores are typically reported as a percentage of the $1M total value earned, and shift by model release. See the [official GitHub repository](https://github.com/openai/SWELancer-Benchmark) and paper for the full per-model breakdown across IC and managerial task splits. ## SWE-Lancer on Benchgen > No Benchgen results yet — [be the first to run SWE-Lancer](/benchmarks/samuel-miserendino-michele-wang-tejal-patwardhan-johannes-heidecke-openai/swe-lancer). ## SWE-Lancer vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **SWE-Lancer** | Real freelance IC + managerial engineering tasks | 1,400+ | low | | [SWE-bench Verified](/benchmarks/swe-bench-verified) | GitHub issue-to-PR resolution | 500 | medium | | [RepoBench](/benchmarks/tianyang-liu-canwen-xu-julian-mcauley-uc-san-diego/repobench) | Repository-level code completion | — | medium | SWE-Lancer is distinguished from SWE-bench Verified by its real freelance-marketplace payout framing and its unique managerial decision-making task type, going beyond pure code-patch resolution. ## Run SWE-Lancer on Your Model Benchgen lets teams evaluate their own coding model or agent on SWE-Lancer-style IC and managerial tasks, tracking both pass rate and dollar-value-earned trends across releases. ## Frequently Asked Questions

What is SWE-Lancer?

SWE-Lancer is an OpenAI benchmark of over 1,400 real freelance software engineering tasks from Upwork, worth $1 million USD total, testing both independent coding ability and managerial decision-making.

What does a good SWE-Lancer score look like?

At publication, frontier models were still unable to resolve the majority of tasks, leaving most of the $1M pool unclaimed — so resolving a meaningful share of high-value tasks represents strong performance.

Who created SWE-Lancer?

SWE-Lancer was created by Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke at OpenAI, published in February 2025.

Rank	Model	Score
1	claude-3-5-sonnet-v1	36.1
2	o1	29.7
3	llama-3-3-70b-instruct	23.5
4	gpt-4o	23.3

SWE-Lancer — Results

SWE-Lancer — Evaluations