AndroidWorld — Results
| Rank | Model | Score |
|---|---|---|
| 1 | qwen3-8-flash-next | 84.5 |
| 2 | qwen3-8-27b | 81.9 |
AndroidWorld
1 phaseActive
A dynamic Android benchmarking environment with 116 parameterized, natural-language tasks across 20 real-world apps, testing autonomous mobile-use agent control. Metric: % task success.
Overview
AndroidWorld
Quick answer: AndroidWorld (Rawles et al., Google DeepMind, 2024) is a fully functional Android environment that dynamically constructs 116 programmatic, natural-language-parameterized tasks across 20 real-world apps to test autonomous agent control. Qwen3.8-Flash-Next scores 84.5% as of October 2026.
At a Glance
What it tests: Whether an autonomous agent can control a real Android device end-to-end — navigating apps, completing multi-step natural-language tasks, and having its success verified by dedicated initialization/checking/tear-down logic that inspects live device state.
Why it matters: Unlike static mobile-agent test sets, AndroidWorld dynamically parameterizes its 116 tasks across 20 real apps in natural language, giving a much larger and more realistic space of mobile-use scenarios than a fixed benchmark — making it a strong proxy for real-world mobile agent reliability.
Known limitations: The original paper's own baseline agents complete only 30.6% of tasks, showing significant headroom remained at release; task variations can meaningfully affect agent performance (a robustness finding from the paper itself), so single-run scores can vary run to run.
What AndroidWorld Measures
AndroidWorld provides a fully functional Android environment with reward signals for 116 tasks spanning 20 real-world Android apps. Unlike static test sets, it dynamically constructs tasks that are parameterized and expressed in natural language in effectively unlimited variations, enabling evaluation on a much larger and more realistic task suite than a fixed benchmark allows. Each task includes dedicated initialization, success-checking, and tear-down logic that modifies and inspects the device's actual system state, so success is verified against ground truth rather than approximated.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Agent (mobile-use/GUI control) |
| Metric | % task success |
| Number of tasks | 116 parameterized tasks across 20 apps |
| Saturation | Low |
| Created by | Rawles et al. (Google DeepMind) |
| Source paper | Rawles et al. 2024 |
| GitHub | google-research/android_world |
How AndroidWorld Is Scored
An agent is scored on the percentage of the 116 tasks it completes successfully, as verified by each task's own initialization/success-checking/tear-down logic against the live Android device state — not by string-matching or static rubric grading.
State-of-the-Art Results
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | Qwen3.8-Flash-Next | 84.5% | Qwen3.8-Flash-Next model card | 2026-08 |
| 2 | Qwen3.8-27B | 81.9% | Qwen3.8-27B model card | 2026-08 |
Scores above are self-reported by Alibaba's Qwen Team on the official Hugging Face model cards and shown for context. They are not Benchgen measurements, and are not drawn from a shared third-party leaderboard — only the 2 scores above are currently tracked locally.
AndroidWorld on Benchgen
No Benchgen results yet — be the first to run AndroidWorld.
AndroidWorld vs Other Benchmarks
| Benchmark | What it tests | Saturation |
|---|---|---|
| AndroidWorld | Dynamic, parameterized mobile-use agent control across 20 real Android apps | Low |
| OSWorld-Verified | Real desktop/web computer-use agent control | — |
AndroidWorld is the mobile-device counterpart to desktop-focused agent benchmarks like OSWorld-Verified — use it when you need to assess an agent's ability to control a real Android device across diverse, dynamically-varied app tasks rather than a static desktop or browser environment.
Run AndroidWorld on Your Model
Benchgen lets you evaluate your own model against AndroidWorld's 116 dynamically-parameterized Android tasks and track results across model versions.
Related Models & Benchmarks
- Top performers: Qwen3.8-Flash-Next · Qwen3.8-27B
- Related benchmarks: OSWorld-Verified
Frequently Asked Questions
What is AndroidWorld?
AndroidWorld is a Google DeepMind benchmark (Rawles et al., 2024) providing a fully functional Android environment with 116 dynamically-parameterized, natural-language tasks across 20 real-world apps, used to evaluate autonomous mobile-use agents.What does a good AndroidWorld score look like?
Qwen3.8-Flash-Next reports 84.5% as of October 2026, the highest currently tracked on Benchgen; Qwen3.8-27B scores 81.9%. The original 2024 paper's own baseline agents completed only 30.6% of tasks, so scores in the 80%+ range reflect substantial progress since release.Who created AndroidWorld?
AndroidWorld was created by Christopher Rawles and co-authors at Google DeepMind; see the original paper.Is AndroidWorld saturated?
No — AndroidWorld remains a low-saturation benchmark; its dynamically-parameterized natural-language tasks and ground-truth device-state verification keep it a meaningful ongoing challenge for mobile-use agents.How does AndroidWorld differ from OSWorld-Verified?
AndroidWorld targets real Android mobile devices and apps, while OSWorld-Verified targets real desktop/web computer-use workflows — both verify success against live environment state rather than static answer-matching, but for different device form factors.Benchmark definition paraphrased from Rawles et al. 2024. State-of-the-art scores sourced from published model cards and attributed inline. Last updated 2026-10-31.