# OSWorld-Verified ![Category](https://img.shields.io/badge/Category-Computer%20Use%20Agent-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Success-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-369-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-Apr%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202404.07972-b31b1b)](https://arxiv.org/abs/2404.07972) [![GitHub](https://img.shields.io/badge/GitHub-xlang--ai%2FOSWorld-181717?logo=github&logoColor=white)](https://github.com/xlang-ai/OSWorld) > **Quick answer:** OSWorld-Verified is a re-validated subset of OSWorld (Xie et al., 2024), a scalable real-computer environment that tests AI agents on 369 open-ended tasks across real Ubuntu, Windows, and macOS desktops — file I/O, multi-app workflows, and web/desktop apps. The "Verified" label refers to task suites with corrected setup configs and evaluation scripts used by third-party evaluators such as Artificial Analysis. Kimi K3 scores 84.8% as of July 2026. ## At a Glance **What it tests:** An agent's ability to operate a real operating system GUI end-to-end — opening applications, navigating file systems, and completing multi-step workflows across office, browser, and creative apps. **Why it matters:** Most computer-use benchmarks rely on simplified sandboxes. OSWorld runs agents against real OS environments with execution-based (not LLM-judged) evaluation scripts, making it one of the most realistic tests of GUI grounding and operational competence. **Known limitations:** Original OSWorld baselines showed a huge human-vs-model gap (humans ~72%, best 2024 models ~12%); by mid-2026 frontier agents have closed much of this gap, so headline numbers should be read against the evaluation date. ## What OSWorld-Verified Measures OSWorld provides an interactive, scalable computer environment supporting task setup, execution-based evaluation, and interactive learning across multiple operating systems. Its 369-task benchmark spans real web and desktop applications, OS-level file operations, and workflows that require chaining multiple applications together — for example, extracting data from a spreadsheet and formatting it into a document. The "Verified" variant re-runs the original task suite with corrected initial-state configurations and reference evaluation scripts, removing ambiguous or broken tasks identified after the original 2024 release. This is the configuration most commonly used by third-party evaluators (including Artificial Analysis) when comparing frontier agents in 2025–2026. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Agent / computer use (GUI) | | Metric | % task success (execution-based) | | Number of tasks | 369 | | Operating systems | Ubuntu, Windows, macOS | | Saturation | Low | | Created by | Xie et al. (HKU, Salesforce Research, CMU, University of Waterloo) | | Source paper | [OSWorld (arXiv 2404.07972)](https://arxiv.org/abs/2404.07972) | | GitHub | [xlang-ai/OSWorld](https://github.com/xlang-ai/OSWorld) | ## How OSWorld-Verified Is Scored Each task ships with a custom execution-based evaluation script that checks the final OS/application state against the expected outcome (e.g., a file's contents, an application setting, or a rendered document). Scores are reported as % of tasks passed, with no partial credit for most tasks. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Qwen3.8 Max](/models/alibaba/qwen3-8-max) | 86.1% | llm-stats.com aggregator | 2026-08 | | 2 | [Claude Fable 5](/models/anthropic/claude-fable-5) | 85.0% | llm-stats.com aggregator | 2026-08 | | 3 | [Kimi K3](/models/moonshot-ai/kimi-k3) | 84.8% | [Kimi K3 technical report](https://www.kimi.com/blog/kimi-k3) | 2026-07 | | 4 | [Claude Opus 4.8](/models/anthropic/claude-opus-4-8) | 83.4% | llm-stats.com aggregator | 2026-08 | | 5 | [Claude Sonnet 5](/models/anthropic/claude-sonnet-5) | 81.2% | llm-stats.com aggregator | 2026-08 | | 6 | [GPT-5.5](/models/openai/gpt-5-5) | 78.7% | llm-stats.com aggregator | 2026-08 | | 7 | [Claude Opus 4.7](/models/anthropic/claude-opus-4-7) | 78.0% | llm-stats.com aggregator | 2026-08 | | 8 | [Kimi K2.6](/models/moonshot/kimi-k2-6) | 73.1% | llm-stats.com aggregator | 2026-08 | | 9 | [Muse Glimmer](/models/meta/muse-glimmer) | 65.9% | Meta model card | 2026-08 | | 10 | [Qwen3.6 Plus](/models/alibaba/qwen3-6-plus) | 62.5% | llm-stats.com aggregator | 2026-08 | *Scores 1, 4, 5, 6, 7, 8, 10 are sourced from the llm-stats.com public aggregator (August 2026 pull) and are not independently verified by Benchgen; Kimi K3 and Muse Glimmer scores are sourced directly from vendor technical reports/model cards. Treat aggregator-sourced scores as directional rather than authoritative — see each model's own page for sourcing detail.* ## OSWorld-Verified on Benchgen > No Benchgen results yet — [be the first to run OSWorld-Verified](/benchmarks/osworld/osworld-verified). ## OSWorld-Verified vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **OSWorld-Verified** | Real desktop/web GUI agent workflows | 369 | Low | | [OSWorld 2.0](/benchmarks/osworld/osworld-20) | Expanded/updated GUI agent task suite | — | Low | | [MCP Atlas](/benchmarks/scale-ai/mcp-atlas) | MCP tool-use agentic workflows | ~500 | Low | | [Tau3 Banking](/benchmarks/sierra-research/3-banking) | Customer-service agentic tasks | ~200 | Low | ## Run OSWorld-Verified on Your Model Benchgen lets you run OSWorld-Verified against your own model and agent harness, version-tracking GUI task success rates so you can catch regressions before they reach production computer-use agents. ## Frequently Asked Questions

What is OSWorld-Verified?

OSWorld-Verified is a corrected, re-validated subset of the OSWorld benchmark (Xie et al., 2024), which tests AI agents on 369 real computer-use tasks across Ubuntu, Windows, and macOS using execution-based evaluation.

What does a good score look like on OSWorld-Verified?

At release, the best 2024 models scored around 12% against a ~72% human baseline. By mid-2026, frontier agents such as Kimi K3 report scores in the 80%+ range, reflecting rapid progress in GUI grounding.

Who created OSWorld?

OSWorld was created by Tianbao Xie and collaborators at the University of Hong Kong, Salesforce Research, CMU, and the University of Waterloo (arXiv:2404.07972, 2024).

Rank	Model	Score
1	qwen3-8-max	86.1
2	claude-fable-5	85
3	kimi-k3	84.8
4	qwen3-8-27b	84.3
5	claude-opus-4-8	83.4
6	nex-n2-5-pro	82.2
7	mimo-v2-6-pro-rl	82
8	claude-sonnet-5	81.2
9	mimo-v2-6-flash-rl	80.8
10	muse-spark-open-816b-a42b	80.4
11	gpt-5-5	78.7
12	claude-opus-4-7	78
13	kimi-k2-6	73.1
14	muse-glimmer	65.9
15	qwen3-6-plus	62.5

OSWorld-Verified — Results

OSWorld-Verified — Phases

OSWorld-Verified