# AndroidWorld ![Category](https://img.shields.io/badge/Category-Agent-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Task%20Success-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-116-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-2024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202405.14573-b31b1b)](https://arxiv.org/abs/2405.14573) [![GitHub](https://img.shields.io/badge/GitHub-google--research%2Fandroid__world-181717?logo=github&logoColor=white)](https://github.com/google-research/android_world) > **Quick answer:** AndroidWorld (Rawles et al., Google DeepMind, 2024) is a fully functional Android environment that dynamically constructs 116 programmatic, natural-language-parameterized tasks across 20 real-world apps to test autonomous agent control. Qwen3.8-Flash-Next scores 84.5% as of October 2026. ## At a Glance **What it tests:** Whether an autonomous agent can control a real Android device end-to-end — navigating apps, completing multi-step natural-language tasks, and having its success verified by dedicated initialization/checking/tear-down logic that inspects live device state. **Why it matters:** Unlike static mobile-agent test sets, AndroidWorld dynamically parameterizes its 116 tasks across 20 real apps in natural language, giving a much larger and more realistic space of mobile-use scenarios than a fixed benchmark — making it a strong proxy for real-world mobile agent reliability. **Known limitations:** The original paper's own baseline agents complete only 30.6% of tasks, showing significant headroom remained at release; task variations can meaningfully affect agent performance (a robustness finding from the paper itself), so single-run scores can vary run to run. ## What AndroidWorld Measures AndroidWorld provides a fully functional Android environment with reward signals for 116 tasks spanning 20 real-world Android apps. Unlike static test sets, it dynamically constructs tasks that are parameterized and expressed in natural language in effectively unlimited variations, enabling evaluation on a much larger and more realistic task suite than a fixed benchmark allows. Each task includes dedicated initialization, success-checking, and tear-down logic that modifies and inspects the device's actual system state, so success is verified against ground truth rather than approximated. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Agent (mobile-use/GUI control) | | Metric | % task success | | Number of tasks | 116 parameterized tasks across 20 apps | | Saturation | Low | | Created by | Rawles et al. (Google DeepMind) | | Source paper | [Rawles et al. 2024](https://arxiv.org/abs/2405.14573) | | GitHub | [google-research/android_world](https://github.com/google-research/android_world) | ## How AndroidWorld Is Scored An agent is scored on the percentage of the 116 tasks it completes successfully, as verified by each task's own initialization/success-checking/tear-down logic against the live Android device state — not by string-matching or static rubric grading. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Qwen3.8-Flash-Next](/models/qwen/qwen3-8-flash-next) | 84.5% | Qwen3.8-Flash-Next model card | 2026-08 | | 2 | [Qwen3.8-27B](/models/qwen/qwen3-8-27b) | 81.9% | Qwen3.8-27B model card | 2026-08 | *Scores above are self-reported by Alibaba's Qwen Team on the official Hugging Face model cards and shown for context. They are not Benchgen measurements, and are not drawn from a shared third-party leaderboard — only the 2 scores above are currently tracked locally.* ## AndroidWorld on Benchgen > No Benchgen results yet — [be the first to run AndroidWorld](/benchmarks/google-deepmind/androidworld). ## AndroidWorld vs Other Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | **AndroidWorld** | Dynamic, parameterized mobile-use agent control across 20 real Android apps | Low | | [OSWorld-Verified](/benchmarks/osworld/osworld-verified) | Real desktop/web computer-use agent control | — | AndroidWorld is the mobile-device counterpart to desktop-focused agent benchmarks like OSWorld-Verified — use it when you need to assess an agent's ability to control a real Android device across diverse, dynamically-varied app tasks rather than a static desktop or browser environment. ## Run AndroidWorld on Your Model Benchgen lets you evaluate your own model against AndroidWorld's 116 dynamically-parameterized Android tasks and track results across model versions. ## Related Models & Benchmarks - Top performers: [Qwen3.8-Flash-Next](/models/qwen/qwen3-8-flash-next) · [Qwen3.8-27B](/models/qwen/qwen3-8-27b) - Related benchmarks: [OSWorld-Verified](/benchmarks/osworld/osworld-verified) ## Frequently Asked Questions

What is AndroidWorld?

AndroidWorld is a Google DeepMind benchmark (Rawles et al., 2024) providing a fully functional Android environment with 116 dynamically-parameterized, natural-language tasks across 20 real-world apps, used to evaluate autonomous mobile-use agents.

What does a good AndroidWorld score look like?

Qwen3.8-Flash-Next reports 84.5% as of October 2026, the highest currently tracked on Benchgen; Qwen3.8-27B scores 81.9%. The original 2024 paper's own baseline agents completed only 30.6% of tasks, so scores in the 80%+ range reflect substantial progress since release.

Who created AndroidWorld?

AndroidWorld was created by Christopher Rawles and co-authors at Google DeepMind; see the [original paper](https://arxiv.org/abs/2405.14573).

Is AndroidWorld saturated?

No — AndroidWorld remains a low-saturation benchmark; its dynamically-parameterized natural-language tasks and ground-truth device-state verification keep it a meaningful ongoing challenge for mobile-use agents.

How does AndroidWorld differ from OSWorld-Verified?

AndroidWorld targets real Android mobile devices and apps, while OSWorld-Verified targets real desktop/web computer-use workflows — both verify success against live environment state rather than static answer-matching, but for different device form factors.

--- *Benchmark definition paraphrased from [Rawles et al. 2024](https://arxiv.org/abs/2405.14573). State-of-the-art scores sourced from published model cards and attributed inline. Last updated 2026-10-31.*

Rank	Model	Score
1	qwen3-8-flash-next	84.5
2	qwen3-8-27b	81.9

AndroidWorld — Results

AndroidWorld — Evaluations