RULER 128K — Results
| Rank | Model | Score |
|---|---|---|
| 1 | granite-4-2-30b | 81.38 |
| 2 | granite-4-2-8b | 71.41 |
| 3 | lfm3-1-7b | 65.56 |
| 4 | granite-4-2-3b | 55.3 |
RULER 128K
1 phaseActive
NVIDIA's synthetic long-context benchmark at the 128K-token length, expanding needle-in-a-haystack retrieval with multi-hop tracing and aggregation tasks.
Overview
RULER 128K
Quick answer: RULER (Hsieh, Sun, Kriman, Acharya, Rekesh, Jia, Zhang & Ginsburg, NVIDIA, COLM 2024) is a synthetic long-context benchmark that expands the classic needle-in-a-haystack (NIAH) retrieval test with multi-hop tracing and aggregation tasks, evaluated across configurable sequence lengths. This page tracks the 128K-token length variant. IBM Granite 4.2 30B scores 81.38% as of August 2026.
At a Glance
What it tests: Whether a model's long-context understanding holds up as sequence length grows to 128K tokens — not just simple retrieval, but variable-tracking across multi-hop chains and aggregation over scattered information.
Why it matters: RULER's own paper found that models claiming 128K+ context routinely show large accuracy drops well before that length. Tracking 128K as its own page (rather than one blended average) makes it possible to see exactly how much a model degrades at the length it actually claims to support.
Known limitations: RULER is entirely synthetic (no real-world documents), so strong RULER scores don't guarantee equivalent performance on real long-document tasks; it also only evaluated 13 task configurations in the original paper, which the authors note "cannot replace more preferred realistic tasks."
What RULER Measures
RULER was built to go beyond the vanilla NIAH test, which the authors show is "indicative of only a superficial form of long-context understanding." It expands NIAH with multiple needle types/quantities, then adds two new task categories the original paper introduced specifically to probe behaviors beyond simple searching: multi-hop tracing (following variable-name binding chains) and aggregation (extracting/counting information spread across the context). Combined with a standard question-answering task category, RULER evaluates 13 task configurations in total across 4 task categories (retrieval, multi-hop tracing, aggregation, QA).
Scores are reported per sequence length (commonly 4K/8K/16K/32K/64K/128K); this page tracks the 128K-token length specifically, since the original paper's central finding is that accuracy degrades as length increases even for models claiming much larger context windows.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Reasoning (long-context retrieval, multi-hop tracing, aggregation) |
| Metric | % accuracy (per sequence length) |
| Number of tasks | 13 task configurations across 4 categories |
| Context length (this page) | 128K tokens |
| Saturation | Low |
| Created by | Hsieh et al. (NVIDIA) |
| Source paper | Hsieh et al. 2024, COLM 2024 |
| GitHub | hsiehjackson/RULER |
How RULER Is Scored
Each of the 13 synthetic tasks is scored for correctness at a given sequence length, then averaged into a per-length accuracy percentage (the figure tracked on this page, at 128K tokens). The original paper additionally reports a length-weighted average (wAvg.) across all tested lengths, but per-length scores — as cited on this page — are the figures models and labs most commonly self-report.
RULER on Benchgen
No Benchgen results yet — be the first to run RULER 128K.
State-of-the-Art Results
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | Granite 4.2 30B | 81.38% | Granite 4.2 launch blog | 2026-08 |
| 2 | Granite 4.2 8B | 71.41% | Granite 4.2 launch blog | 2026-08 |
| 3 | LFM3-1.7B | 65.56% | LFM3-1.7B model card | 2026-10 |
| 4 | Granite 4.2 3B | 55.30% | Granite 4.2 launch blog | 2026-08 |
Scores above are self-reported by IBM Research and Liquid AI on their respective official model cards/launch blogs and shown for context. They are not Benchgen measurements, and are not drawn from a shared third-party leaderboard — only the 4 scores above are currently tracked locally.
RULER vs Other Long-Context Benchmarks
| Benchmark | What it tests | Saturation |
|---|---|---|
| RULER | Synthetic long-context retrieval, multi-hop tracing, aggregation, per sequence length | Low |
| MRCRv2 | Multi-turn conversation retrieval correctness | Low |
| LongBench v2 | Realistic long-document multi-task understanding | Low |
| AA-LCR | Long-context reasoning across documents | Low |
RULER is purely synthetic and length-parameterized, which makes it well-suited for isolating exactly where a model's long-context performance starts to degrade — unlike realistic-document benchmarks like LongBench v2, which blend retrieval with task-specific reasoning over genuine long texts.
Run RULER on Your Model
Benchgen lets you evaluate your own model against RULER's synthetic long-context suite at 128K tokens and track results across model versions.
Related Models & Benchmarks
- Top performers: Granite 4.2 30B · Granite 4.2 8B · LFM3-1.7B · Granite 4.2 3B
- Related benchmarks: RULER 64K · MRCRv2 · LongBench v2
Frequently Asked Questions
What is RULER?
RULER is a synthetic long-context benchmark introduced by Hsieh, Sun, Kriman, Acharya, Rekesh, Jia, Zhang & Ginsburg at NVIDIA (COLM 2024), expanding needle-in-a-haystack retrieval with multi-hop tracing and aggregation tasks across 13 task configurations and configurable sequence lengths.What does a good RULER 128K score look like?
IBM's Granite 4.2 30B reports 81.38% as of August 2026, the highest currently tracked on Benchgen at this length; Liquid AI's LFM3-1.7B scores 65.56% despite being a much smaller 1.73B-parameter edge model.Who created RULER?
RULER was created by Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg at NVIDIA; see the original paper, published at COLM 2024.Why does RULER have separate pages per context length?
RULER's central finding is that model accuracy degrades as sequence length increases, so a single blended score would hide that pattern; Benchgen splits RULER into per-length pages (64K, 128K) the same way it splits BrowseComp Long Context into 128K/256K variant pages, so scores stay comparable at a fixed length.Is RULER saturated?
No — the original paper found that even models claiming 128K+ token context windows often show large accuracy drops well before that length, confirming RULER remains a low-saturation benchmark at 128K.Benchmark definition paraphrased from Hsieh et al. 2024, COLM 2024. State-of-the-art scores sourced from published model cards/blogs and attributed inline. Last updated 2026-10-31.