# RULER 64K ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-13-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-Apr%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202404.06654-b31b1b)](https://arxiv.org/abs/2404.06654) [![GitHub](https://img.shields.io/badge/GitHub-hsiehjackson%2FRULER-181717?logo=github&logoColor=white)](https://github.com/hsiehjackson/RULER) > **Quick answer:** RULER (Hsieh, Sun, Kriman, Acharya, Rekesh, Jia, Zhang & Ginsburg, NVIDIA, COLM 2024) is a synthetic long-context benchmark that expands the classic needle-in-a-haystack (NIAH) retrieval test with multi-hop tracing and aggregation tasks, evaluated across configurable sequence lengths. This page tracks the **64K-token** length variant. IBM Granite 4.2 30B scores 89.96% as of August 2026. ## At a Glance **What it tests:** Whether a model's long-context understanding holds up as sequence length grows — not just simple retrieval, but variable-tracking across multi-hop chains and aggregation over scattered information, at a fixed 64K-token context length. **Why it matters:** Plain needle-in-a-haystack tests are easy to saturate; RULER's own paper found that models claiming 32K+ context routinely show large accuracy drops well before that length, and only about half could maintain satisfactory performance even at 32K. Per-length pages like this one make it possible to compare models at a specific context length rather than only an averaged score. **Known limitations:** RULER is entirely synthetic (no real-world documents), so strong RULER scores don't guarantee equivalent performance on real long-document tasks; it also only evaluated 13 task configurations in the original paper, which the authors note "cannot replace more preferred realistic tasks." ## What RULER Measures RULER was built to go beyond the vanilla NIAH test, which the authors show is "indicative of only a superficial form of long-context understanding." It expands NIAH with multiple needle types/quantities, then adds two new task categories the original paper introduced specifically to probe behaviors beyond simple searching: **multi-hop tracing** (following variable-name binding chains) and **aggregation** (extracting/counting information spread across the context). Combined with a standard question-answering task category, RULER evaluates 13 task configurations in total across 4 task categories (retrieval, multi-hop tracing, aggregation, QA). Scores are reported per sequence length (commonly 4K/8K/16K/32K/64K/128K), since the original paper's central finding is that accuracy degrades as length increases even for models claiming much larger context windows — a single averaged score across lengths would hide that degradation curve. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning (long-context retrieval, multi-hop tracing, aggregation) | | Metric | % accuracy (per sequence length) | | Number of tasks | 13 task configurations across 4 categories | | Context length (this page) | 64K tokens | | Saturation | Low | | Created by | Hsieh et al. (NVIDIA) | | Source paper | [Hsieh et al. 2024, COLM 2024](https://arxiv.org/abs/2404.06654) | | GitHub | [hsiehjackson/RULER](https://github.com/hsiehjackson/RULER) | ## How RULER Is Scored Each of the 13 synthetic tasks is scored for correctness at a given sequence length, then averaged into a per-length accuracy percentage (the figure tracked on this page, at 64K tokens). The original paper additionally reports a length-weighted average (wAvg.) across all tested lengths, but per-length scores — as cited on this page — are the figures models and labs most commonly self-report. ## RULER on Benchgen > No Benchgen results yet — [be the first to run RULER 64K](/benchmarks/nvidia/ruler-64k). ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Granite 4.2 30B](/models/ibm/granite-4-2-30b) | 89.96% | [Granite 4.2 launch blog](https://huggingface.co/blog/ibm-granite/granite-4-2) | 2026-08 | | 2 | [Granite 4.2 8B](/models/ibm/granite-4-2-8b) | 80.99% | [Granite 4.2 launch blog](https://huggingface.co/blog/ibm-granite/granite-4-2) | 2026-08 | | 3 | [Granite 4.2 3B](/models/ibm/granite-4-2-3b) | 67.52% | [Granite 4.2 launch blog](https://huggingface.co/blog/ibm-granite/granite-4-2) | 2026-08 | *Scores above are self-reported by IBM Research on the official Granite 4.2 Hugging Face launch blog and shown for context. They are not Benchgen measurements, and are not drawn from a shared third-party leaderboard — only the 3 scores above are currently tracked locally.* ## RULER vs Other Long-Context Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | **RULER** | Synthetic long-context retrieval, multi-hop tracing, aggregation, per sequence length | Low | | [MRCRv2](/benchmarks/google-deepmind/mrcrv2) | Multi-turn conversation retrieval correctness | Low | | [LongBench v2](/benchmarks/longbench-v2) | Realistic long-document multi-task understanding | Low | | [AA-LCR](/benchmarks/artificial-analysis/aa-lcr) | Long-context reasoning across documents | Low | RULER is purely synthetic and length-parameterized, which makes it well-suited for isolating exactly where a model's long-context performance starts to degrade — unlike realistic-document benchmarks like LongBench v2, which blend retrieval with task-specific reasoning over genuine long texts. ## Run RULER on Your Model Benchgen lets you evaluate your own model against RULER's synthetic long-context suite at 64K tokens and track results across model versions. ## Related Models & Benchmarks - Top performers: [Granite 4.2 30B](/models/ibm/granite-4-2-30b) · [Granite 4.2 8B](/models/ibm/granite-4-2-8b) · [Granite 4.2 3B](/models/ibm/granite-4-2-3b) - Related benchmarks: [RULER 128K](/benchmarks/nvidia/ruler-128k) · [MRCRv2](/benchmarks/google-deepmind/mrcrv2) · [LongBench v2](/benchmarks/longbench-v2) ## Frequently Asked Questions

What is RULER?

RULER is a synthetic long-context benchmark introduced by Hsieh, Sun, Kriman, Acharya, Rekesh, Jia, Zhang & Ginsburg at NVIDIA (COLM 2024), expanding needle-in-a-haystack retrieval with multi-hop tracing and aggregation tasks across 13 task configurations and configurable sequence lengths.

What does a good RULER 64K score look like?

IBM's Granite 4.2 30B reports 89.96% as of August 2026, the highest currently tracked on Benchgen at this length; Granite 4.2 8B scores 80.99% and Granite 4.2 3B scores 67.52%.

Who created RULER?

RULER was created by Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg at NVIDIA; see the [original paper](https://arxiv.org/abs/2404.06654), published at COLM 2024.

Why does RULER have separate pages per context length?

RULER's central finding is that model accuracy degrades as sequence length increases, so a single blended score would hide that pattern; Benchgen splits RULER into per-length pages (64K, 128K) the same way it splits BrowseComp Long Context into 128K/256K variant pages, so scores stay comparable at a fixed length.

Is RULER saturated?

No — the original paper found that even models claiming 32K+ token context windows often show large accuracy drops well before that length, and only about half maintained satisfactory performance at 32K, confirming RULER remains a low-saturation benchmark.

--- *Benchmark definition paraphrased from [Hsieh et al. 2024](https://arxiv.org/abs/2404.06654), COLM 2024. State-of-the-art scores sourced from published model cards/blogs and attributed inline. Last updated 2026-10-31.*

Rank	Model	Score
1	granite-4-2-30b	89.96
2	granite-4-2-8b	80.99
3	granite-4-2-3b	67.52

RULER 64K — Results

RULER 64K — Evaluations