ScreenSpot
1 phaseActive
GUI grounding benchmark across mobile, desktop, and web screenshots, created by researchers at Nanjing University and Shanghai AI Lab for the SeeClick agent.
Overview
ScreenSpot
Quick answer: ScreenSpot is a realistic GUI grounding benchmark introduced alongside the SeeClick visual GUI agent, testing whether a model can locate the correct screen element (icon, button, text field) from a natural-language instruction across mobile, desktop, and web screenshots.
At a Glance
What it tests: GUI grounding — mapping a natural-language instruction ("click the search icon") to the correct pixel location on a real app or website screenshot, without relying on HTML/accessibility-tree structure.
Why it matters: Screenshot-only grounding is a prerequisite capability for general-purpose visual GUI agents that must operate on any interface, including ones without accessible structured data (e.g., many desktop apps).
Known limitations: ScreenSpot's static screenshots test single-step grounding, not multi-step task completion, and coverage of niche or highly customized UIs is limited relative to the diversity of real-world software.
What ScreenSpot Measures
ScreenSpot was introduced in the SeeClick paper as the first realistic GUI grounding benchmark spanning three environments — mobile, desktop, and web — collected from real applications and websites rather than synthetic UI mockups. For each screenshot, an instruction describes a target element, and the model must predict a click point or bounding box; a prediction counts as correct only if it falls within the ground-truth element's region.
The benchmark's authors found that GUI grounding performance directly correlates with downstream GUI agent task success, motivating ScreenSpot's use as a fast, single-step proxy for evaluating full agentic capability without running expensive multi-step task rollouts.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Multimodal (GUI grounding) |
| Metric | Click/grounding accuracy (%) — correct if predicted point/box falls within the target element |
| Number of tasks | 1,272 instructions across mobile, desktop, and web screenshots |
| Saturation | Medium — specialized GUI-grounding models outperform general-purpose VLMs, with headroom on desktop/web |
| Created by | Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, Zhiyong Wu |
| Source paper | Cheng et al. 2024 |
| Code & data | GitHub |
How ScreenSpot Is Scored
Each instruction is paired with a target element's ground-truth bounding box. A model's predicted click coordinate (or box) is scored correct if it falls inside the target region; accuracy is reported overall and broken down by platform (mobile/desktop/web) and element type (text vs. icon/widget).
State-of-the-Art Results
Scores fluctuate by model release cadence and evaluation harness (prompt template, screenshot resolution). See the official repository for current standings across mobile, desktop, and web splits.
ScreenSpot on Benchgen
No Benchgen results yet — be the first to run ScreenSpot.
ScreenSpot vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| ScreenSpot | Single-step GUI element grounding | 1,272 | medium |
| VisualWebBench | Broader web-page understanding + grounding | — | medium |
| RefCOCOg | Natural-image object grounding | — | high |
ScreenSpot's GUI-specific grounding complements RefCOCOg's natural-image object grounding, and pairs naturally with broader web-understanding benchmarks like VisualWebBench when evaluating GUI agents end-to-end.
Run ScreenSpot on Your Model
Benchgen lets teams run ScreenSpot's mobile, desktop, and web splits against their own model versions on demand, tracking grounding accuracy by platform and element type to identify GUI-agent readiness gaps.
Related Models & Benchmarks
Explore related GUI-agent benchmarks like VisualWebBench and grounding benchmarks like RefCOCOg to compare GUI-specific and general-purpose visual grounding evaluation approaches.