# MMSearch ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Weighted%20Search%20Score-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-300-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Sep%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2409.12959) [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/CaraJ7/MMSearch) > **Quick answer:** MMSearch evaluates whether Large Multimodal Models can act as effective multimodal AI search engines, using the MMSearch-Engine pipeline to test models across requery, rerank, summarization, and full end-to-end searching tasks on 300 curated queries with no training-data overlap. ## At a Glance **What it tests:** Multimodal search capability — reformulating a query (with image and/or text), reranking retrieved website results, summarizing findings, and completing the full end-to-end search process to answer a question correctly. **Why it matters:** AI search engines had largely remained text-only; MMSearch is one of the first benchmarks to systematically evaluate whether LMMs can handle multimodal, text-image-interleaved search — directly relevant as multimodal AI search products (like Perplexity and SearchGPT) proliferate. **Known limitations:** With only 300 instances, MMSearch trades scale for careful construction (ensuring no training-data overlap), and end-to-end evaluation depends on live web search results, which can vary run-to-run based on search engine behavior. ## What MMSearch Measures MMSearch first introduces MMSearch-Engine, a pipeline that equips any LMM with multimodal search capability, then evaluates models on top of it using MMSearch — a benchmark of 300 manually collected instances spanning 14 subfields, specifically curated to have no overlap with LMMs' training data so correct answers can only come from genuine searching. Models are assessed on three individual sub-tasks — requery (reformulating the search query), rerank (ordering retrieved results by relevance), and summarization (synthesizing an answer from retrieved content) — plus one challenging end-to-end task that chains the full search process together. At release, GPT-4o paired with MMSearch-Engine achieved the best results among tested models, notably surpassing the commercial product Perplexity Pro on the end-to-end task. Error analysis in the paper found that current LMMs still struggle to fully grasp multimodal search tasks, and ablations suggested that scaling test-time computation offers a promising path to improving AI search engine performance — directly relevant to teams building agentic, tool-using search products. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (AI search engine evaluation) | | Metric | Weighted composite score across requery, rerank, summarization, and end-to-end tasks | | Number of tasks | 300 manually collected instances across 14 subfields | | Saturation | Low — even GPT-4o + MMSearch-Engine leaves substantial room for improvement | | Created by | Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, and collaborators | | Source paper | [Jiang et al. 2024](https://arxiv.org/abs/2409.12959) (ICLR 2025) | | Code | [GitHub](https://github.com/CaraJ7/MMSearch) | | Dataset | [Hugging Face](https://huggingface.co/datasets/CaraJ/MMSearch) | ## How MMSearch Is Scored Each of the four tasks (requery, rerank, summarization, end-to-end) is scored individually against ground truth, then combined into a final weighted score; the requery task is automatically evaluated as part of the end-to-end task, so the three components teams typically report are end-to-end, rerank, and summarization. ## State-of-the-Art Results > Scores vary by underlying LMM and search-engine configuration. See the [official leaderboard](https://mmsearch.github.io/#leaderboard) for current standings. ## MMSearch on Benchgen > No Benchgen results yet — [be the first to run MMSearch](/benchmarks/jiang-et-al-cuhk-mmlab/mmsearch). ## MMSearch vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **MMSearch** | Multimodal AI search-engine capability (requery/rerank/summarize/end-to-end) | 300 | low | | [MMT-Bench](/benchmarks/ying-et-al-shanghai-ai-laboratory/mmt-bench) | 32-meta-task multitask multimodal benchmark | 31,325 | medium | | [MTVQA](/benchmarks/tang-et-al-bytedance/mtvqa) | Multilingual text-centric visual QA | 6,778 | low | MMSearch is distinctive among multimodal benchmarks in testing an end-to-end agentic search workflow rather than a single-turn QA format, making it directly relevant to teams building search-augmented agents. ## Run MMSearch on Your Model Benchgen lets teams evaluate their own model or agent stack on the full MMSearch pipeline on demand, tracking requery, rerank, summarization, and end-to-end scores over time to catch regressions in agentic search quality. ## Related Models & Benchmarks Explore related multimodal benchmarks like [MMT-Bench](/benchmarks/ying-et-al-shanghai-ai-laboratory/mmt-bench) and [MTVQA](/benchmarks/tang-et-al-bytedance/mtvqa) to compare broad multitask and multilingual evaluation with MMSearch's agentic search focus. ## Frequently Asked Questions

What is MMSearch?

MMSearch is a benchmark evaluating whether Large Multimodal Models can function as multimodal AI search engines, testing requery, rerank, summarization, and end-to-end search tasks on 300 curated instances.

What does a good MMSearch score look like?

At release, GPT-4o paired with the MMSearch-Engine pipeline achieved the best results among tested models and surpassed Perplexity Pro on the end-to-end task, but the paper's error analysis found substantial room for improvement remains across all models.

Who created MMSearch?

MMSearch was created by Dongzhi Jiang and collaborators at CUHK MMLab and Shanghai AI Laboratory, accepted at ICLR 2025.

Rank	Model	Score
1	gpt-4o	62.3
2	gpt-4v	55
3	claude-3-5-sonnet	53.5
4	llava-onevision-72b	50.1

MMSearch — Results

MMSearch — Phases

MMSearch