MMSearch — Results
| Rank | Model | Score |
|---|---|---|
| 1 | gpt-4o | 62.3 |
| 2 | gpt-4v | 55 |
| 3 | claude-3-5-sonnet | 53.5 |
| 4 | llava-onevision-72b | 50.1 |
MMSearch
1 phaseActive
300 manually curated multimodal search instances across 14 subfields, evaluating requery, rerank, summarization, and end-to-end search tasks.
Overview
MMSearch
Quick answer: MMSearch evaluates whether Large Multimodal Models can act as effective multimodal AI search engines, using the MMSearch-Engine pipeline to test models across requery, rerank, summarization, and full end-to-end searching tasks on 300 curated queries with no training-data overlap.
At a Glance
What it tests: Multimodal search capability — reformulating a query (with image and/or text), reranking retrieved website results, summarizing findings, and completing the full end-to-end search process to answer a question correctly.
Why it matters: AI search engines had largely remained text-only; MMSearch is one of the first benchmarks to systematically evaluate whether LMMs can handle multimodal, text-image-interleaved search — directly relevant as multimodal AI search products (like Perplexity and SearchGPT) proliferate.
Known limitations: With only 300 instances, MMSearch trades scale for careful construction (ensuring no training-data overlap), and end-to-end evaluation depends on live web search results, which can vary run-to-run based on search engine behavior.
What MMSearch Measures
MMSearch first introduces MMSearch-Engine, a pipeline that equips any LMM with multimodal search capability, then evaluates models on top of it using MMSearch — a benchmark of 300 manually collected instances spanning 14 subfields, specifically curated to have no overlap with LMMs' training data so correct answers can only come from genuine searching.
Models are assessed on three individual sub-tasks — requery (reformulating the search query), rerank (ordering retrieved results by relevance), and summarization (synthesizing an answer from retrieved content) — plus one challenging end-to-end task that chains the full search process together. At release, GPT-4o paired with MMSearch-Engine achieved the best results among tested models, notably surpassing the commercial product Perplexity Pro on the end-to-end task.
Error analysis in the paper found that current LMMs still struggle to fully grasp multimodal search tasks, and ablations suggested that scaling test-time computation offers a promising path to improving AI search engine performance — directly relevant to teams building agentic, tool-using search products.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Multimodal (AI search engine evaluation) |
| Metric | Weighted composite score across requery, rerank, summarization, and end-to-end tasks |
| Number of tasks | 300 manually collected instances across 14 subfields |
| Saturation | Low — even GPT-4o + MMSearch-Engine leaves substantial room for improvement |
| Created by | Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, and collaborators |
| Source paper | Jiang et al. 2024 (ICLR 2025) |
| Code | GitHub |
| Dataset | Hugging Face |
How MMSearch Is Scored
Each of the four tasks (requery, rerank, summarization, end-to-end) is scored individually against ground truth, then combined into a final weighted score; the requery task is automatically evaluated as part of the end-to-end task, so the three components teams typically report are end-to-end, rerank, and summarization.
State-of-the-Art Results
Scores vary by underlying LMM and search-engine configuration. See the official leaderboard for current standings.
MMSearch on Benchgen
No Benchgen results yet — be the first to run MMSearch.
MMSearch vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| MMSearch | Multimodal AI search-engine capability (requery/rerank/summarize/end-to-end) | 300 | low |
| MMT-Bench | 32-meta-task multitask multimodal benchmark | 31,325 | medium |
| MTVQA | Multilingual text-centric visual QA | 6,778 | low |
MMSearch is distinctive among multimodal benchmarks in testing an end-to-end agentic search workflow rather than a single-turn QA format, making it directly relevant to teams building search-augmented agents.
Run MMSearch on Your Model
Benchgen lets teams evaluate their own model or agent stack on the full MMSearch pipeline on demand, tracking requery, rerank, summarization, and end-to-end scores over time to catch regressions in agentic search quality.
Related Models & Benchmarks
Explore related multimodal benchmarks like MMT-Bench and MTVQA to compare broad multitask and multilingual evaluation with MMSearch's agentic search focus.