IFEval — Results
| Rank | Model | Score |
|---|---|---|
| 1 | lfm2-5-24b-a2b | 96.75 |
| 2 | lfm3-1-7b | 96.66 |
| 3 | flint | 95.1 |
| 4 | spark-x2-5-4b | 93 |
| 5 | spark-x2-5-4b | 93 |
| 6 | twil-8b | 91.8 |
| 7 | spark-x2-5-1-7b | 89.5 |
| 8 | spark-x2-5-1-7b | 89.5 |
| 9 | minicpm5-2b | 86.7 |
| 10 | lfm2-5-vl-3b | 82.3 |
| 11 | north-micro-vision-instruct | 74.9 |
| 12 | qwen3-5-27b | 0.95 |
| 13 | qwen3-7-plus | 0.946 |
| 14 | qwen3-6-plus | 0.943 |
| 15 | qwen3-7-max | 0.943 |
| 16 | o3-mini | 0.939 |
| 17 | qwen3-5-122b-a10b | 0.934 |
| 18 | qwen3-5-397b-a17b | 0.926 |
| 19 | llama-3-3-70b-instruct | 0.921 |
| 20 | qwen3-5-35b-a3b | 0.919 |
| 21 | qwen3-5-9b | 0.915 |
| 22 | kimi-k2-instruct-0905 | 0.898 |
| 23 | kimi-k2-instruct | 0.898 |
| 24 | qwen3-5-4b | 0.898 |
| 25 | llama-3-1-nemotron-ultra-253b-v1 | 0.894 |
IFEval
1 phaseActive
Instruction-Following Evaluation — 541 prompts with verifiable constraints (format, length, keyword, case). Metric: prompt-level and instruction-level accuracy.
Overview
IFEval
Quick answer: IFEval (Instruction-Following Evaluation) is a benchmark from Google Research (Zhou et al., 2023) that evaluates language models on their ability to follow verifiable, precise instructions. It contains 541 prompts with 25 types of verifiable constraints — such as word count limits, keyword inclusion, format requirements, and capitalization rules. Unlike open-ended instruction benchmarks, IFEval constraints can be checked programmatically, making it deterministic and reproducible.
At a Glance
What it tests: Whether models precisely follow explicit, measurable constraints in their responses — length limits, required keywords, response format (JSON, bullet lists, markdown), forbidden words, and letter-case rules.
Why it matters: IFEval directly measures a model's ability to comply with system-level and user-level formatting requirements — a critical capability for production applications, APIs, and structured output use cases. It complements benchmark suites focused on knowledge or reasoning.
Known limitations: IFEval is increasingly saturated for frontier models (scores above 85–90%). It tests rule-following but not semantic quality of the response. A model could score highly while producing unhelpful content that technically satisfies the constraints.
What IFEval Measures
IFEval is built on 541 prompts covering diverse instruction types. Each prompt includes one or more verifiable constraints drawn from 25 categories including:
- Format constraints: respond in JSON, use markdown headers, use numbered lists
- Length constraints: respond in exactly N words, fewer than N sentences
- Keyword constraints: include specific words, avoid certain words
- Case constraints: respond in all caps, start each sentence with a capital letter
- Content constraints: include a specific phrase, answer in a particular language
The benchmark reports two primary metrics:
- Prompt-level accuracy: percentage of prompts where ALL constraints are satisfied
- Instruction-level accuracy: percentage of individual constraint instances satisfied
Benchmark Specifications
| Field | Value |
|---|---|
| Prompts | 541 |
| Constraint types | 25 |
| Primary metric | Prompt-level accuracy |
| Secondary metric | Instruction-level accuracy |
| Evaluation | Programmatic (deterministic) |
| Created by | Zhou et al. (Google Research) |
| Paper | arXiv:2311.07911 (Nov 2023) |
| Saturation | High — frontier models 85–95% |
Frequently Asked Questions
How is IFEval different from IFBench? IFEval uses programmatically verifiable constraints and was created by Google Research (2023). IFBench is a separate benchmark with a broader, more complex set of multi-constraint instruction-following tasks.
What is a good IFEval score? Frontier models (GPT-4o, Claude 3.5+, Gemini 1.5+) score 80–90%+ on prompt-level accuracy. Weaker models often score below 60%. The benchmark is approaching saturation at the frontier.
Does IFEval correlate with real-world instruction following? IFEval correlates with structured output reliability, but it tests only explicit constraint-following, not semantic helpfulness. Models that score well may still fail on implicit or context-dependent instructions.