# OSWorld 2.0 ![Category](https://img.shields.io/badge/Category-Computer%20Use%20Agent-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Success-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-2026-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202404.07972-b31b1b)](https://arxiv.org/abs/2404.07972) [![GitHub](https://img.shields.io/badge/GitHub-xlang--ai%2FOSWorld-181717?logo=github&logoColor=white)](https://github.com/xlang-ai/OSWorld) > **Quick answer:** OSWorld 2.0 is an expanded, more difficult revision of the original OSWorld computer-use agent benchmark, broadening application coverage and task complexity beyond the original 369-task suite. Kimi K3 scores 58.3%, notably lower than its 84.8% on OSWorld-Verified — reflecting the harder task pool introduced in the 2.0 revision. ## At a Glance **What it tests:** Real desktop and web GUI task completion, with a broader and harder task pool than the original OSWorld release — including longer workflows and more diverse applications. **Why it matters:** As agents saturate the original OSWorld task suite, OSWorld 2.0 provides continued headroom to differentiate frontier computer-use agents. **Known limitations:** As a newer revision, less third-party leaderboard data exists compared to the original OSWorld/OSWorld-Verified suite, so cross-model comparisons are still limited. ## What OSWorld 2.0 Measures OSWorld 2.0 builds on the original OSWorld environment — an interactive, scalable real-computer sandbox spanning Ubuntu, Windows, and macOS — with an expanded and more challenging task pool. Tasks continue to use execution-based evaluation (checking final system/application state) rather than LLM-as-judge scoring, preserving OSWorld's reproducibility advantage while raising the difficulty ceiling as frontier agents improve. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Agent / computer use (GUI) | | Metric | % task success (execution-based) | | Operating systems | Ubuntu, Windows, macOS | | Saturation | Low | | Based on | OSWorld (Xie et al., arXiv 2404.07972) | | GitHub | [xlang-ai/OSWorld](https://github.com/xlang-ai/OSWorld) | ## How OSWorld 2.0 Is Scored Tasks are scored pass/fail via execution-based scripts that verify the resulting OS or application state matches the expected outcome. Aggregate scores are reported as % of tasks completed successfully. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Kimi K3](/models/moonshot/kimi-k3) | 58.3% | [Kimi K3 technical report](https://www.kimi.com/blog/kimi-k3) | 2026-07 | *Score sourced from Moonshot AI's Kimi K3 announcement, July 2026.* ## OSWorld 2.0 on Benchgen > No Benchgen results yet — [be the first to run OSWorld 2.0](/benchmarks/osworld-2). ## OSWorld 2.0 vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | **OSWorld 2.0** | Expanded, harder GUI agent workflows | Low | | [OSWorld-Verified](/benchmarks/osworld-verified) | Original, corrected GUI agent task suite | Low | | [MCP Atlas](/benchmarks/mcp-atlas) | MCP tool-use agentic workflows | Low | | [SaaS-Bench](/benchmarks/saas-bench) | SaaS product usage workflows | Low | ## Run OSWorld 2.0 on Your Model Benchgen lets you run OSWorld 2.0 against your own model and harness configuration, tracking GUI task success over time as you iterate on agent scaffolding and fine-tuning. ## Frequently Asked Questions

What is OSWorld 2.0?

OSWorld 2.0 is an expanded, harder revision of the OSWorld computer-use agent benchmark, testing AI agents on real desktop and web GUI tasks with broader application coverage than the original release.

How does OSWorld 2.0 differ from OSWorld-Verified?

OSWorld-Verified is a corrected version of the original 369-task OSWorld suite. OSWorld 2.0 introduces a broader, more difficult task pool, which is why models like Kimi K3 score notably lower on 2.0 (58.3%) than on Verified (84.8%).

Who maintains OSWorld?

OSWorld originates from Tianbao Xie and collaborators at HKU, Salesforce Research, CMU, and the University of Waterloo. Subsequent revisions, including 2.0, are maintained by the same open-source project.

OSWorld 2.0 — Results

OSWorld 2.0 — Phases

OSWorld 2.0