MGSM — Results
| Rank | Model | Score |
|---|---|---|
| 1 | llama-4-maverick | 0.923 |
| 2 | o3-mini | 0.92 |
| 3 | claude-3-5-sonnet | 0.916 |
| 4 | llama-3-3-70b-instruct | 0.911 |
| 5 | llama-4-scout | 0.906 |
| 6 | gpt-4o | 0.905 |
| 7 | o1 | 0.893 |
| 8 | gpt-4-turbo | 0.885 |
| 9 | gemini-1-5-pro | 0.875 |
| 10 | gpt-4o-mini | 0.87 |
| 11 | llama-3-2-90b-instruct | 0.869 |
| 12 | claude-3-5-haiku | 0.856 |
| 13 | qwen3-235b-a22b | 0.835 |
| 14 | gemini-1-5-flash | 0.826 |
| 15 | phi-4 | 0.806 |
| 16 | phi-4-mini | 0.639 |
| 17 | phi-3-5-moe-instruct | 0.587 |
| 18 | llama-3-2-3b-instruct | 0.582 |
| 19 | phi-3-5-mini-instruct | 0.479 |
MGSM
1 phaseActive
Multilingual Grade School Math — 250 problems per language across 10 languages translated from GSM8K. Tests mathematical reasoning across language boundaries. Metric: accuracy.
Overview
MGSM
Quick answer: MGSM (Multilingual Grade School Math) is a multilingual extension of GSM8K created by Shi et al. (2022). It contains 250 problems per language across 10 languages — Bengali, Chinese, French, German, Japanese, Russian, Spanish, Swahili, Telugu, and Thai — for a total of 2,500 problems. It tests whether language models can solve grade-school math word problems when presented in languages other than English, revealing cross-lingual reasoning transfer.
At a Glance
What it tests: Mathematical reasoning in 10 diverse languages. Problems are direct translations of GSM8K problems, controlling for content difficulty while varying only the language. This isolates the model's ability to understand and reason in non-English languages.
Why it matters: MGSM is a critical benchmark for multilingual model evaluation — particularly for understanding whether math reasoning degrades in low-resource languages like Swahili and Telugu. It reveals large gaps between English performance and performance in other languages, especially for smaller models.
Known limitations: Saturated for frontier multilingual models (85–95% on major languages). Translations maintain GSM8K's difficulty, which is relatively easy for modern models. Performance variance is highest for low-resource languages.
What MGSM Measures
MGSM uses 250 problems from the GSM8K test set, professionally translated into 10 languages:
| Language | Script | Resource Level |
|---|---|---|
| Bengali | বাংলা | Medium |
| Chinese (Simplified) | 中文 | High |
| French | Français | High |
| German | Deutsch | High |
| Japanese | 日本語 | High |
| Russian | Русский | High |
| Spanish | Español | High |
| Swahili | Kiswahili | Low |
| Telugu | తెలుగు | Low |
| Thai | ภาษาไทย | Medium |
The primary metric is accuracy averaged across all 10 languages. Results are also commonly broken down per language.
Benchmark Specifications
| Field | Value |
|---|---|
| Problems | 2,500 (250 per language) |
| Languages | 10 |
| Source | GSM8K (250-problem subset) |
| Primary metric | Accuracy (avg across languages) |
| Created by | Shi et al. |
| Paper | arXiv:2210.01362 (Oct 2022) |
| Saturation | High for major languages, medium for low-resource |
Frequently Asked Questions
How does MGSM compare to GSM8K? MGSM uses the same 250 problems from GSM8K translated into 10 languages. GSM8K uses English only with 1,319 test problems. MGSM specifically tests multilingual generalization, while GSM8K is the standard English math benchmark.
What score do frontier models get on MGSM? Frontier models (GPT-4o, Claude 3.5+, Gemini 1.5+) score 85–95% averaged across all 10 languages. Performance in Swahili and Telugu is typically 5–15 points lower than in French, German, or Chinese.
Is MGSM still useful for evaluation? MGSM remains useful for tracking multilingual capability gaps between languages. While saturated on major languages, it still discriminates models on low-resource languages. New models are often still evaluated on MGSM for comparability with historical results.