JFBench

JFBench

Quick answer: JFBench (Preferred Networks) is a Japanese-language instruction-following benchmark built in the IFBench style — prompts carry 1, 2, 4, or 8 simultaneous constraints, and responses are scored by an LLM judge (by default gpt-oss-120b) for whether every constraint was satisfied. It's designed to test whether a model's instruction-following reliability, well-studied in English, holds up in Japanese.

At a Glance

What it tests: Whether an LLM can satisfy multiple simultaneous, verifiable constraints (formatting, content, style) in a single Japanese-language response, at increasing constraint counts (1, 2, 4, 8).

Why it matters: Instruction-following benchmarks are overwhelmingly English-first; JFBench specifically probes whether that reliability transfers to Japanese, where honorific register, sentence structure, and script mixing (kanji/hiragana/katakana) introduce failure modes that don't exist in English.

Known limitations: Uses an LLM (gpt-oss-120b by default) rather than fully deterministic rule-based verification for judging some constraints, and is a relatively new, smaller-community benchmark (MIT-licensed, ~15 GitHub stars as of mid-2026) compared to established English IF benchmarks.

What JFBench Measures

JFBench extends the IFBench methodology — prompts constructed from a pool of programmatically checkable constraints — into Japanese, evaluating models at four constraint-count tiers (1, 2, 4, and 8 constraints layered into a single prompt). As constraint count rises, a model must simultaneously satisfy more requirements in one response, which sharply increases difficulty: models that handle a single constraint reliably often fail as more are stacked together.

Because natural Japanese has features that don't map cleanly onto English instruction templates (e.g. honorific/polite registers, particle usage, script choice), JFBench serves as a check on whether a model's English instruction-following score is representative of its real-world Japanese usability, or whether Japanese-specific tuning is needed to close the gap.

Benchmark Specifications

How JFBench Is Scored

For each prompt, the constraints attached to it (1, 2, 4, or 8 depending on the test tier) are each checked against the model's response; a response only counts as fully correct if every attached constraint is satisfied. An LLM judge (gpt-oss-120b by default, configurable) performs the constraint checks not resolvable by simple string/format rules. Overall score is the share of prompts where all constraints were satisfied, aggregated across constraint-count tiers.

JFBench on Benchgen

No Benchgen results yet — be the first to run JFBench.

JFBench vs Other Benchmarks

JFBench is the right choice for teams specifically validating Japanese-language product reliability; IFBench and Multi-IF are better for general English or broad multilingual instruction-following comparisons.

FAQ

What is JFBench? JFBench is Preferred Networks' benchmark for Japanese instruction-following, testing whether models can satisfy 1, 2, 4, or 8 simultaneous constraints in a single Japanese-language response.

How is JFBench scored? Each response is checked against all constraints attached to its prompt; scoring some constraints requires an LLM judge (default gpt-oss-120b). The overall score is the percentage of prompts where every constraint was satisfied.

Who created JFBench? Preferred Networks, released as an open-source (MIT-licensed) tool at github.com/pfnet-research/jfbench.

Is JFBench saturated? No — scores in the mid-30s to high-50s (%) for current frontier and specialized models indicate significant headroom remains, especially at higher constraint counts.

How does JFBench differ from IFBench? IFBench tests English-language instruction following with held-out (novel) instruction types; JFBench applies a similar multi-constraint methodology specifically in Japanese, at fixed constraint-count tiers (1/2/4/8).

Benchmark definition paraphrased from the pfnet-research/jfbench README. Last updated 2026-08-05.