IFBench

IFBench

Quick answer: IFBench is an instruction-following generalization benchmark from the Allen Institute for AI (Pyatkin et al., NeurIPS 2025) that evaluates how well language models comply with 58 novel, out-of-domain verifiable output constraints — such as "mention the word X at least N times" or "only answer with yes or no" — applied to real WildChat prompts. It is explicitly designed to test generalisation to unseen constraints, not memorised IFEval patterns. Nemotron 3 Ultra leads among the Inkling comparison set at 81.4%.

At a Glance

What it tests: A model's ability to precisely follow novel, verifiable output constraints it has not seen during training — covering format, length, word inclusion, position, and other structural requirements applied to real-world conversational prompts.

Why it matters: Most models overfit to the small constraint vocabulary in IFEval (the dominant prior benchmark), masking whether they have genuine instruction-following capability or just pattern-matched to training examples. IFBench's 58 out-of-domain constraints expose that gap: models that look similar on IFEval often diverge significantly on IFBench. It was accepted to NeurIPS 2025 Datasets & Benchmarks.

Known limitations: 58 constraint types is a relatively small evaluation surface. Scores are sensitive to output extraction method, particularly for thinking/reasoning models where the reasoning chain must be stripped before grading. The paper reports "prompt-level loose accuracy" — a single failure on any constraint in a prompt counts as a miss.

What IFBench Measures

IFBench pairs real user prompts from WildChat (a large collection of real ChatGPT conversations) with 58 new verifiable output constraints that do not appear in the IFEval training distribution. Constraints span categories like:

• Format constraints — respond in a specific structure, use bullet points, write in a particular style
• Length constraints — answer in under N words, write exactly N sentences
• Lexical constraints — include/exclude specific words or phrases, mention a keyword N times
• Position constraints — start/end the response with a specific token or phrase

Each constraint has a corresponding automated verification function, making scoring fully programmatic and reproducible without a judge model. The benchmark is evaluated at prompt-level loose accuracy: a prompt is counted as correct only if the model's response satisfies the constraint; partial credit is not given.

The benchmark also includes a two-turn multiturn variant where the constraint is issued in a second turn, testing whether models can retroactively apply constraints to a completed response.

Benchmark Specifications

How IFBench Is Scored

Scores are reported as prompt-level loose accuracy (% of prompts where the model's response satisfies the constraint). Each constraint type is paired with a deterministic verification function — no LLM judge is used. Higher scores indicate better generalisation to novel output constraints.

For thinking/reasoning models, the reasoning chain is extracted and stripped before grading so only the final response is evaluated. Temperature 0 is used for generation. The paper notes that thinking models generally benefit from higher max-token budgets to allow the chain-of-thought before the final answer.

State-of-the-Art Results

Scores from Inkling model card (Thinking Machines Lab, July 2026), evaluated at effort=0.99.

Note: Claude Fable 5's unusually low score relative to its performance on other benchmarks may reflect sensitivity to constraint formats or output extraction — consistent with IFBench's design goal of exposing generalisation failures.

IFBench vs Related Benchmarks

Last updated 2026-07-16.