# ScreenSpot ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Grounding%20Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-1%2C272-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-medium-d29922) ![Created](https://img.shields.io/badge/Created-Jan%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2401.10935) [![GitHub](https://img.shields.io/badge/GitHub-Repo-181717?logo=github&logoColor=white)](https://github.com/njucckevin/SeeClick) > **Quick answer:** ScreenSpot is a realistic GUI grounding benchmark introduced alongside the SeeClick visual GUI agent, testing whether a model can locate the correct screen element (icon, button, text field) from a natural-language instruction across mobile, desktop, and web screenshots. ## At a Glance **What it tests:** GUI grounding — mapping a natural-language instruction ("click the search icon") to the correct pixel location on a real app or website screenshot, without relying on HTML/accessibility-tree structure. **Why it matters:** Screenshot-only grounding is a prerequisite capability for general-purpose visual GUI agents that must operate on any interface, including ones without accessible structured data (e.g., many desktop apps). **Known limitations:** ScreenSpot's static screenshots test single-step grounding, not multi-step task completion, and coverage of niche or highly customized UIs is limited relative to the diversity of real-world software. ## What ScreenSpot Measures ScreenSpot was introduced in the SeeClick paper as the first realistic GUI grounding benchmark spanning three environments — mobile, desktop, and web — collected from real applications and websites rather than synthetic UI mockups. For each screenshot, an instruction describes a target element, and the model must predict a click point or bounding box; a prediction counts as correct only if it falls within the ground-truth element's region. The benchmark's authors found that GUI grounding performance directly correlates with downstream GUI agent task success, motivating ScreenSpot's use as a fast, single-step proxy for evaluating full agentic capability without running expensive multi-step task rollouts. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (GUI grounding) | | Metric | Click/grounding accuracy (%) — correct if predicted point/box falls within the target element | | Number of tasks | 1,272 instructions across mobile, desktop, and web screenshots | | Saturation | Medium — specialized GUI-grounding models outperform general-purpose VLMs, with headroom on desktop/web | | Created by | Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, Zhiyong Wu | | Source paper | [Cheng et al. 2024](https://arxiv.org/abs/2401.10935) | | Code & data | [GitHub](https://github.com/njucckevin/SeeClick) | ## How ScreenSpot Is Scored Each instruction is paired with a target element's ground-truth bounding box. A model's predicted click coordinate (or box) is scored correct if it falls inside the target region; accuracy is reported overall and broken down by platform (mobile/desktop/web) and element type (text vs. icon/widget). ## State-of-the-Art Results > Scores fluctuate by model release cadence and evaluation harness (prompt template, screenshot resolution). See the [official repository](https://github.com/njucckevin/SeeClick) for current standings across mobile, desktop, and web splits. ## ScreenSpot on Benchgen > No Benchgen results yet — [be the first to run ScreenSpot](/benchmarks/kanzhi-cheng-et-al-nanjing-university-shanghai-ai-lab/screenspot). ## ScreenSpot vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **ScreenSpot** | Single-step GUI element grounding | 1,272 | medium | | [VisualWebBench](/benchmarks/visualwebbench) | Broader web-page understanding + grounding | — | medium | | [RefCOCOg](/benchmarks/refcocog) | Natural-image object grounding | — | high | ScreenSpot's GUI-specific grounding complements RefCOCOg's natural-image object grounding, and pairs naturally with broader web-understanding benchmarks like VisualWebBench when evaluating GUI agents end-to-end. ## Run ScreenSpot on Your Model Benchgen lets teams run ScreenSpot's mobile, desktop, and web splits against their own model versions on demand, tracking grounding accuracy by platform and element type to identify GUI-agent readiness gaps. ## Related Models & Benchmarks Explore related GUI-agent benchmarks like [VisualWebBench](/benchmarks/visualwebbench) and grounding benchmarks like [RefCOCOg](/benchmarks/refcocog) to compare GUI-specific and general-purpose visual grounding evaluation approaches. ## Frequently Asked Questions

What is ScreenSpot?

ScreenSpot is a GUI grounding benchmark with 1,272 instructions across mobile, desktop, and web screenshots, testing whether a model can locate the correct on-screen element from a natural-language instruction.

What does a good ScreenSpot score look like?

Specialized GUI-grounding models can exceed 80% accuracy overall, while general-purpose multimodal LLMs without GUI-specific training often score well below 50%, especially on desktop and web splits.

Who created ScreenSpot?

ScreenSpot was created by Kanzhi Cheng and collaborators at Nanjing University and Shanghai AI Laboratory, introduced alongside the SeeClick visual GUI agent.

ScreenSpot — Phases

ScreenSpot