# MTVQA ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-6%2C778-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-May%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2405.11985) [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/bytedance/MTVQA) > **Quick answer:** MTVQA is the first multilingual text-centric visual question answering benchmark built with high-quality human expert annotations, covering 9 widely-used but low-resource languages (Arabic, German, French, Italian, Japanese, Korean, Russian, Thai, Vietnamese) to test genuine multilingual scene-text comprehension. ## At a Glance **What it tests:** Reading and reasoning over visual text (signs, documents, UI, labels) embedded in images across 9 non-English, non-Chinese languages — a scenario most prior text-centric VQA benchmarks skip. **Why it matters:** Earlier multilingual VQA efforts relied on machine-translating English question-answer pairs, which causes a "visual-textual misalignment" problem — the translated text no longer matches the actual scene text in the image; MTVQA avoids this with native human annotation in each target language. **Known limitations:** Coverage is limited to 9 languages, so it does not capture the full diversity of world scripts and languages, and scene-text OCR difficulty varies significantly across these languages (e.g., Thai and Japanese remain especially hard). ## What MTVQA Measures MTVQA features 6,778 question-answer pairs across 2,116 images, with high-quality human expert annotations spanning 9 diverse languages. Unlike translation-based multilingual VQA construction, which prioritizes the text in QA pairs while disregarding the actual visual text in images, MTVQA's native-language annotation avoids the resulting misalignment and better captures nuanced meaning, contextual distortion, language bias, and diverse question types. At release, the paper found substantial headroom for improvement even among state-of-the-art MLLMs — Qwen2-VL scored 30.9 versus a 79.7 human baseline — establishing MTVQA as a meaningfully unsaturated multilingual text-comprehension benchmark. The dataset also supplies multilingual training data, and the authors demonstrated that straightforward fine-tuning on this data substantially improves multilingual TEC-VQA performance. MTVQA has been rapidly adopted as a standard multilingual OCR/VQA reference point, with continuous leaderboard updates as new frontier and open-source MLLMs are tested (InternVL2.5, Qwen2-VL 72B, BlueLM-V-3B, and others). ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (multilingual text-centric visual QA) | | Metric | Accuracy (%), reported per language and averaged overall | | Number of tasks | 6,778 QA pairs across 2,116 images in 9 languages | | Saturation | Low — best models still trail human accuracy by a wide margin | | Created by | Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, and collaborators | | Source paper | [Tang et al. 2024](https://arxiv.org/abs/2405.11985) (ACL 2025 Findings) | | Code | [GitHub](https://github.com/bytedance/MTVQA) | | Dataset | [Hugging Face](https://huggingface.co/datasets/ByteDance/MTVQA) | ## How MTVQA Is Scored Each language subset is scored as accuracy against ground-truth answers, then reported both per-language (AR, DE, FR, IT, JA, KO, RU, TH, VI) and as an overall average, letting teams identify language-specific weaknesses (e.g., a model strong in European languages but weak in Thai or Japanese). ## State-of-the-Art Results | Rank | Model | Score (AVG) | Source | |---|---|---|---| | 1 | InternVL2.5 78B | 31.9 | [MTVQA leaderboard](https://github.com/bytedance/MTVQA) | | 2 | Helpy-V Reasoning | 31.6 | [MTVQA leaderboard](https://github.com/bytedance/MTVQA) | | 3 | Qwen2-VL 72B | 30.9 | [MTVQA leaderboard](https://github.com/bytedance/MTVQA) | *Scores sourced from the official MTVQA leaderboard; see the repository for the full per-language breakdown.* ## MTVQA on Benchgen > No Benchgen results yet — [be the first to run MTVQA](/benchmarks/tang-et-al-bytedance/mtvqa). ## MTVQA vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **MTVQA** | Multilingual text-centric visual QA (9 languages) | 6,778 | low | | [MME](/benchmarks/chaoyou-fu-et-al-xiamen-university-tencent-youtu-lab/mme) | Broad perception + cognition, yes/no questions | 2,374 | high | | [MMSearch](/benchmarks/jiang-et-al-cuhk-mmlab/mmsearch) | Multimodal search-engine reasoning | 300 | low | MTVQA fills a gap that broad multimodal benchmarks like MME largely skip — genuine multilingual scene-text comprehension — making it a key reference for teams targeting non-English deployments. ## Run MTVQA on Your Model Benchgen lets teams run MTVQA against their own model or agent versions on demand, tracking per-language accuracy over time to catch multilingual regressions that an English-only evaluation suite would miss entirely. ## Related Models & Benchmarks Explore related multimodal benchmarks like [MME](/benchmarks/chaoyou-fu-et-al-xiamen-university-tencent-youtu-lab/mme) and [MMSearch](/benchmarks/jiang-et-al-cuhk-mmlab/mmsearch) to compare broad perception and search-engine reasoning with MTVQA's multilingual text-comprehension focus. ## Frequently Asked Questions

What is MTVQA?

MTVQA is the first multilingual text-centric visual question answering benchmark with human expert annotations across 9 low-resource languages, testing whether MLLMs can read and reason over visual text in non-English scripts.

What does a good MTVQA score look like?

At release, the best models scored around 30-32 average accuracy versus a 79.7 human baseline, so any model materially above the low-30s average represents a meaningfully strong result.

Who created MTVQA?

MTVQA was created by Jingqun Tang and collaborators at ByteDance, accepted at ACL 2025 Findings.

MTVQA — Results

MTVQA — Evaluations