RULER 64K — Results
| Rank | Model | Score |
|---|---|---|
| 1 | granite-4-2-30b | 89.96 |
| 2 | granite-4-2-8b | 80.99 |
| 3 | granite-4-2-3b | 67.52 |
RULER 64K
1 phaseActive
NVIDIA's synthetic long-context benchmark at the 64K-token length, expanding needle-in-a-haystack retrieval with multi-hop tracing and aggregation tasks.
Overview
RULER 64K
Quick answer: RULER (Hsieh, Sun, Kriman, Acharya, Rekesh, Jia, Zhang & Ginsburg, NVIDIA, COLM 2024) is a synthetic long-context benchmark that expands the classic needle-in-a-haystack (NIAH) retrieval test with multi-hop tracing and aggregation tasks, evaluated across configurable sequence lengths. This page tracks the 64K-token length variant. IBM Granite 4.2 30B scores 89.96% as of August 2026.
At a Glance
What it tests: Whether a model's long-context understanding holds up as sequence length grows — not just simple retrieval, but variable-tracking across multi-hop chains and aggregation over scattered information, at a fixed 64K-token context length.
Why it matters: Plain needle-in-a-haystack tests are easy to saturate; RULER's own paper found that models claiming 32K+ context routinely show large accuracy drops well before that length, and only about half could maintain satisfactory performance even at 32K. Per-length pages like this one make it possible to compare models at a specific context length rather than only an averaged score.
Known limitations: RULER is entirely synthetic (no real-world documents), so strong RULER scores don't guarantee equivalent performance on real long-document tasks; it also only evaluated 13 task configurations in the original paper, which the authors note "cannot replace more preferred realistic tasks."
What RULER Measures
RULER was built to go beyond the vanilla NIAH test, which the authors show is "indicative of only a superficial form of long-context understanding." It expands NIAH with multiple needle types/quantities, then adds two new task categories the original paper introduced specifically to probe behaviors beyond simple searching: multi-hop tracing (following variable-name binding chains) and aggregation (extracting/counting information spread across the context). Combined with a standard question-answering task category, RULER evaluates 13 task configurations in total across 4 task categories (retrieval, multi-hop tracing, aggregation, QA).
Scores are reported per sequence length (commonly 4K/8K/16K/32K/64K/128K), since the original paper's central finding is that accuracy degrades as length increases even for models claiming much larger context windows — a single averaged score across lengths would hide that degradation curve.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Reasoning (long-context retrieval, multi-hop tracing, aggregation) |
| Metric | % accuracy (per sequence length) |
| Number of tasks | 13 task configurations across 4 categories |
| Context length (this page) | 64K tokens |
| Saturation | Low |
| Created by | Hsieh et al. (NVIDIA) |
| Source paper | Hsieh et al. 2024, COLM 2024 |
| GitHub | hsiehjackson/RULER |
How RULER Is Scored
Each of the 13 synthetic tasks is scored for correctness at a given sequence length, then averaged into a per-length accuracy percentage (the figure tracked on this page, at 64K tokens). The original paper additionally reports a length-weighted average (wAvg.) across all tested lengths, but per-length scores — as cited on this page — are the figures models and labs most commonly self-report.
RULER on Benchgen
No Benchgen results yet — be the first to run RULER 64K.
State-of-the-Art Results
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | Granite 4.2 30B | 89.96% | Granite 4.2 launch blog | 2026-08 |
| 2 | Granite 4.2 8B | 80.99% | Granite 4.2 launch blog | 2026-08 |
| 3 | Granite 4.2 3B | 67.52% | Granite 4.2 launch blog | 2026-08 |
Scores above are self-reported by IBM Research on the official Granite 4.2 Hugging Face launch blog and shown for context. They are not Benchgen measurements, and are not drawn from a shared third-party leaderboard — only the 3 scores above are currently tracked locally.
RULER vs Other Long-Context Benchmarks
| Benchmark | What it tests | Saturation |
|---|---|---|
| RULER | Synthetic long-context retrieval, multi-hop tracing, aggregation, per sequence length | Low |
| MRCRv2 | Multi-turn conversation retrieval correctness | Low |
| LongBench v2 | Realistic long-document multi-task understanding | Low |
| AA-LCR | Long-context reasoning across documents | Low |
RULER is purely synthetic and length-parameterized, which makes it well-suited for isolating exactly where a model's long-context performance starts to degrade — unlike realistic-document benchmarks like LongBench v2, which blend retrieval with task-specific reasoning over genuine long texts.
Run RULER on Your Model
Benchgen lets you evaluate your own model against RULER's synthetic long-context suite at 64K tokens and track results across model versions.
Related Models & Benchmarks
- Top performers: Granite 4.2 30B · Granite 4.2 8B · Granite 4.2 3B
- Related benchmarks: RULER 128K · MRCRv2 · LongBench v2
Frequently Asked Questions
What is RULER?
RULER is a synthetic long-context benchmark introduced by Hsieh, Sun, Kriman, Acharya, Rekesh, Jia, Zhang & Ginsburg at NVIDIA (COLM 2024), expanding needle-in-a-haystack retrieval with multi-hop tracing and aggregation tasks across 13 task configurations and configurable sequence lengths.What does a good RULER 64K score look like?
IBM's Granite 4.2 30B reports 89.96% as of August 2026, the highest currently tracked on Benchgen at this length; Granite 4.2 8B scores 80.99% and Granite 4.2 3B scores 67.52%.Who created RULER?
RULER was created by Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg at NVIDIA; see the original paper, published at COLM 2024.Why does RULER have separate pages per context length?
RULER's central finding is that model accuracy degrades as sequence length increases, so a single blended score would hide that pattern; Benchgen splits RULER into per-length pages (64K, 128K) the same way it splits BrowseComp Long Context into 128K/256K variant pages, so scores stay comparable at a fixed length.Is RULER saturated?
No — the original paper found that even models claiming 32K+ token context windows often show large accuracy drops well before that length, and only about half maintained satisfactory performance at 32K, confirming RULER remains a low-saturation benchmark.Benchmark definition paraphrased from Hsieh et al. 2024, COLM 2024. State-of-the-art scores sourced from published model cards/blogs and attributed inline. Last updated 2026-10-31.