Global-MMLU-Lite

Global-MMLU-Lite

Quick answer: Global-MMLU-Lite is a multilingual evaluation benchmark by Cohere Labs (Singh et al., 2024) that tests AI models on broad academic knowledge across 23 languages using 9,200 human-translated 4-choice questions drawn from the original MMLU dataset. It annotates questions as Culturally Sensitive or Culturally Agnostic to surface linguistic and cultural biases. Claude Fable 5 leads at 93.3% with GPT-5.6 Sol close behind at 91.8%.

At a Glance

What it tests: Multilingual academic knowledge across 23 languages — the same MMLU subject domains (STEM, humanities, social sciences, etc.) evaluated in the language the model is expected to serve, not just in English.

Why it matters: Most benchmark evaluations are English-only, masking large capability gaps in multilingual models. Global-MMLU-Lite makes language coverage measurable and comparable: a model that scores 92% in English but 65% in Arabic reveals a practical deployment problem that English-only benchmarks would miss.

Known limitations: Scores are averaged across languages; per-language breakdown may reveal significant variance. The 4-choice format inherited from MMLU carries some saturation risk for frontier models. Cultural annotations are based on regional collaborator judgments and may not be exhaustive.

What Global-MMLU-Lite Measures

Global-MMLU-Lite takes the original MMLU question bank — 14,000 questions across academic disciplines — and provides human-translated, professionally verified versions in 23 languages. For each language, the dataset includes 200 Culturally Sensitive (CS) questions (where the correct answer or framing may vary by cultural context) and 200 Culturally Agnostic (CA) questions (where the correct answer is universal). This 400-question-per-language structure makes cross-language comparisons fair and balanced.

The "Lite" designation distinguishes it from the larger Global-MMLU dataset (42 languages, 590K rows), which uses a mix of machine translations alongside professional ones. Global-MMLU-Lite is fully human-translated or post-edited — making it higher fidelity for evaluation, though more limited in language count.

Scores are reported as overall accuracy (% correct) averaged across all languages and both cultural subsets. Labs may also report per-language or per-subset breakdowns to diagnose specific gaps.

Benchmark Specifications

Languages Covered (v3.0)

Arabic, Bengali, Chinese (Simplified), Czech, Dutch, French, German, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Oriya, Polish, Portuguese, Russian, Slovak, Spanish, Swahili, Tajik, Ukrainian, Welsh (+ English for CA/CS annotation reference).

How Global-MMLU-Lite Is Scored

A model is presented with a question in a target language and must select the correct answer from four options (A–D). Scores are reported as accuracy (% correct). The benchmark is typically evaluated in a 0-shot or 5-shot setting. Reported scores often reflect the average across all 23 languages; per-language breakdowns are available from the dataset but less commonly reported in model cards.

Frontier models tend to score in the high-80s to low-90s on Global-MMLU-Lite as of mid-2026, with closed-weight models leading. Performance gaps between English and low-resource languages (e.g. Swahili, Oriya) are typically 10–20 percentage points even for frontier models.

State-of-the-Art Results

Scores from Inkling model card (Thinking Machines Lab, July 2026), evaluated at effort=0.99.

Global-MMLU-Lite vs Related Benchmarks

Last updated 2026-07-16.