Multi-IF

Multi-IF

Quick answer: Multi-IF (He et al., Meta, Oct 2024) extends IFEval to multi-turn and multilingual settings — 4,501 three-turn conversations, with English prompts translated into 7 additional languages. It measures how well models sustain accurate instruction-following as conversations progress and as language changes, and finds that accuracy degrades with each additional turn for every model tested.

At a Glance

What it tests: Instruction-following accuracy across 3-turn conversations in 8 languages (English plus 7 translations), extending single-turn IFEval-style verifiable instructions into a multi-turn, multilingual setting.

Why it matters: Real-world assistant use is conversational and often not in English. Multi-IF exposes a failure mode invisible to single-turn, English-only instruction benchmarks: accuracy drops turn-over-turn, and non-Latin-script languages (Hindi, Russian, Chinese) show higher error rates.

Known limitations: Uses a hybrid LLM + human annotation pipeline rather than fully deterministic verification for every instruction type, and covers only 8 languages.

What Multi-IF Measures

Multi-IF builds on IFEval's approach of using programmatically verifiable instructions (e.g. "respond in exactly 3 paragraphs," "include the word X at least twice") but extends it in two directions that single-turn benchmarks miss: multi-turn dialogue, where instructions from earlier turns must still be respected in later turns, and multilinguality, where the same prompts are translated into 7 additional languages beyond English. The result is 4,501 conversations, each exactly 3 turns long.

The benchmark's headline finding is that instruction-following reliability decays with conversation length — the paper reports o1-preview dropping from 0.877 accuracy at turn 1 to 0.707 by turn 3, averaged across languages — and that models are measurably worse at following instructions in non-Latin-script languages (Hindi, Russian, Chinese) than in English or other Latin-script languages. This makes Multi-IF a more realistic stress test for production chat assistants than single-turn, English-only instruction benchmarks like the original IFEval.

Benchmark Specifications

How Multi-IF Is Scored

Each conversation contains a sequence of 3 turns, each adding or layering a new verifiable instruction on top of prior ones. A model's response at each turn is checked against the accumulated set of active instructions using a mix of programmatic verifiers (inherited from IFEval) and LLM/human-assisted checks for instructions that aren't purely rule-based. Scores are reported per-turn (showing degradation as the conversation lengthens) and as an overall average across all turns and languages.

Multi-IF on Benchgen

No Benchgen results yet — be the first to run Multi-IF.

Multi-IF vs Other Benchmarks

Multi-IF is the right choice when evaluating conversational assistants that must sustain instruction adherence across turns and languages; IFBench and IFEval remain useful for isolating single-turn, English-only instruction-following capability.

FAQ

What is Multi-IF? Multi-IF is Meta's benchmark for evaluating LLMs on multi-turn, multilingual instruction following — 4,501 three-turn conversations in English plus 7 translated languages, extending the single-turn IFEval methodology.

What does a good Multi-IF score look like? Because accuracy decays with each turn, a "good" score should be read per-turn: strong models score above 0.85 at turn 1 but often drop below 0.75 by turn 3. Non-Latin-script languages typically score lower than English.

Who created Multi-IF? He et al. at Meta, published on arXiv in October 2024 (arXiv:2410.15553).

Is Multi-IF saturated? No — the paper's own results show every one of the 14 models tested degraded across turns, indicating meaningful headroom remains, especially in non-English, later-turn settings.

How does Multi-IF differ from IFBench? IFBench tests single-turn, English-only instruction following with a focus on held-out (novel) instruction types; Multi-IF tests whether that same reliability holds up across 3-turn conversations and 8 languages.

Benchmark definition paraphrased from He et al. 2024 (arXiv:2410.15553). Last updated 2026-08-04.