OSWorld-Verified — Results
| Rank | Model | Score |
|---|---|---|
| 1 | qwen3-8-max | 86.1 |
| 2 | claude-fable-5 | 85 |
| 3 | kimi-k3 | 84.8 |
| 4 | claude-opus-4-8 | 83.4 |
| 5 | claude-sonnet-5 | 81.2 |
| 6 | gpt-5-5 | 78.7 |
| 7 | claude-opus-4-7 | 78 |
| 8 | kimi-k2-6 | 73.1 |
| 9 | muse-glimmer | 65.9 |
| 10 | qwen3-6-plus | 62.5 |
OSWorld-Verified
1 phaseActive
Re-validated subset of OSWorld's real computer-use tasks across Ubuntu, Windows, and macOS — tests GUI grounding and multi-app workflows. Metric: % success.
Overview
OSWorld-Verified
Quick answer: OSWorld-Verified is a re-validated subset of OSWorld (Xie et al., 2024), a scalable real-computer environment that tests AI agents on 369 open-ended tasks across real Ubuntu, Windows, and macOS desktops — file I/O, multi-app workflows, and web/desktop apps. The "Verified" label refers to task suites with corrected setup configs and evaluation scripts used by third-party evaluators such as Artificial Analysis. Kimi K3 scores 84.8% as of July 2026.
At a Glance
What it tests: An agent's ability to operate a real operating system GUI end-to-end — opening applications, navigating file systems, and completing multi-step workflows across office, browser, and creative apps.
Why it matters: Most computer-use benchmarks rely on simplified sandboxes. OSWorld runs agents against real OS environments with execution-based (not LLM-judged) evaluation scripts, making it one of the most realistic tests of GUI grounding and operational competence.
Known limitations: Original OSWorld baselines showed a huge human-vs-model gap (humans ~72%, best 2024 models ~12%); by mid-2026 frontier agents have closed much of this gap, so headline numbers should be read against the evaluation date.
What OSWorld-Verified Measures
OSWorld provides an interactive, scalable computer environment supporting task setup, execution-based evaluation, and interactive learning across multiple operating systems. Its 369-task benchmark spans real web and desktop applications, OS-level file operations, and workflows that require chaining multiple applications together — for example, extracting data from a spreadsheet and formatting it into a document.
The "Verified" variant re-runs the original task suite with corrected initial-state configurations and reference evaluation scripts, removing ambiguous or broken tasks identified after the original 2024 release. This is the configuration most commonly used by third-party evaluators (including Artificial Analysis) when comparing frontier agents in 2025–2026.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Agent / computer use (GUI) |
| Metric | % task success (execution-based) |
| Number of tasks | 369 |
| Operating systems | Ubuntu, Windows, macOS |
| Saturation | Low |
| Created by | Xie et al. (HKU, Salesforce Research, CMU, University of Waterloo) |
| Source paper | OSWorld (arXiv 2404.07972) |
| GitHub | xlang-ai/OSWorld |
How OSWorld-Verified Is Scored
Each task ships with a custom execution-based evaluation script that checks the final OS/application state against the expected outcome (e.g., a file's contents, an application setting, or a rendered document). Scores are reported as % of tasks passed, with no partial credit for most tasks.
State-of-the-Art Results
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | Kimi K3 | 84.8% | Kimi K3 technical report | 2026-07 |
Score sourced from Moonshot AI's Kimi K3 announcement, July 2026.
OSWorld-Verified on Benchgen
No Benchgen results yet — be the first to run OSWorld-Verified.
OSWorld-Verified vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| OSWorld-Verified | Real desktop/web GUI agent workflows | 369 | Low |
| OSWorld 2.0 | Expanded/updated GUI agent task suite | — | Low |
| MCP Atlas | MCP tool-use agentic workflows | ~500 | Low |
| Tau3 Banking | Customer-service agentic tasks | ~200 | Low |
Run OSWorld-Verified on Your Model
Benchgen lets you run OSWorld-Verified against your own model and agent harness, version-tracking GUI task success rates so you can catch regressions before they reach production computer-use agents.