# RULER 128K ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-13-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-Apr%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202404.06654-b31b1b)](https://arxiv.org/abs/2404.06654) [![GitHub](https://img.shields.io/badge/GitHub-hsiehjackson%2FRULER-181717?logo=github&logoColor=white)](https://github.com/hsiehjackson/RULER) > **Quick answer:** RULER (Hsieh, Sun, Kriman, Acharya, Rekesh, Jia, Zhang & Ginsburg, NVIDIA, COLM 2024) is a synthetic long-context benchmark that expands the classic needle-in-a-haystack (NIAH) retrieval test with multi-hop tracing and aggregation tasks, evaluated across configurable sequence lengths. This page tracks the **128K-token** length variant. IBM Granite 4.2 30B scores 81.38% as of August 2026. ## At a Glance **What it tests:** Whether a model's long-context understanding holds up as sequence length grows to 128K tokens — not just simple retrieval, but variable-tracking across multi-hop chains and aggregation over scattered information. **Why it matters:** RULER's own paper found that models claiming 128K+ context routinely show large accuracy drops well before that length. Tracking 128K as its own page (rather than one blended average) makes it possible to see exactly how much a model degrades at the length it actually claims to support. **Known limitations:** RULER is entirely synthetic (no real-world documents), so strong RULER scores don't guarantee equivalent performance on real long-document tasks; it also only evaluated 13 task configurations in the original paper, which the authors note "cannot replace more preferred realistic tasks." ## What RULER Measures RULER was built to go beyond the vanilla NIAH test, which the authors show is "indicative of only a superficial form of long-context understanding." It expands NIAH with multiple needle types/quantities, then adds two new task categories the original paper introduced specifically to probe behaviors beyond simple searching: **multi-hop tracing** (following variable-name binding chains) and **aggregation** (extracting/counting information spread across the context). Combined with a standard question-answering task category, RULER evaluates 13 task configurations in total across 4 task categories (retrieval, multi-hop tracing, aggregation, QA). Scores are reported per sequence length (commonly 4K/8K/16K/32K/64K/128K); this page tracks the 128K-token length specifically, since the original paper's central finding is that accuracy degrades as length increases even for models claiming much larger context windows. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning (long-context retrieval, multi-hop tracing, aggregation) | | Metric | % accuracy (per sequence length) | | Number of tasks | 13 task configurations across 4 categories | | Context length (this page) | 128K tokens | | Saturation | Low | | Created by | Hsieh et al. (NVIDIA) | | Source paper | [Hsieh et al. 2024, COLM 2024](https://arxiv.org/abs/2404.06654) | | GitHub | [hsiehjackson/RULER](https://github.com/hsiehjackson/RULER) | ## How RULER Is Scored Each of the 13 synthetic tasks is scored for correctness at a given sequence length, then averaged into a per-length accuracy percentage (the figure tracked on this page, at 128K tokens). The original paper additionally reports a length-weighted average (wAvg.) across all tested lengths, but per-length scores — as cited on this page — are the figures models and labs most commonly self-report. ## RULER on Benchgen > No Benchgen results yet — [be the first to run RULER 128K](/benchmarks/nvidia/ruler-128k). ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Granite 4.2 30B](/models/ibm/granite-4-2-30b) | 81.38% | [Granite 4.2 launch blog](https://huggingface.co/blog/ibm-granite/granite-4-2) | 2026-08 | | 2 | [Granite 4.2 8B](/models/ibm/granite-4-2-8b) | 71.41% | [Granite 4.2 launch blog](https://huggingface.co/blog/ibm-granite/granite-4-2) | 2026-08 | | 3 | [LFM3-1.7B](/models/liquid-ai/lfm3-1-7b) | 65.56% | [LFM3-1.7B model card](https://huggingface.co/LiquidAI/LFM3-1.7B) | 2026-10 | | 4 | [Granite 4.2 3B](/models/ibm/granite-4-2-3b) | 55.30% | [Granite 4.2 launch blog](https://huggingface.co/blog/ibm-granite/granite-4-2) | 2026-08 | *Scores above are self-reported by IBM Research and Liquid AI on their respective official model cards/launch blogs and shown for context. They are not Benchgen measurements, and are not drawn from a shared third-party leaderboard — only the 4 scores above are currently tracked locally.* ## RULER vs Other Long-Context Benchmarks | Benchmark | What it tests | Saturation | |---|---|---| | **RULER** | Synthetic long-context retrieval, multi-hop tracing, aggregation, per sequence length | Low | | [MRCRv2](/benchmarks/google-deepmind/mrcrv2) | Multi-turn conversation retrieval correctness | Low | | [LongBench v2](/benchmarks/longbench-v2) | Realistic long-document multi-task understanding | Low | | [AA-LCR](/benchmarks/artificial-analysis/aa-lcr) | Long-context reasoning across documents | Low | RULER is purely synthetic and length-parameterized, which makes it well-suited for isolating exactly where a model's long-context performance starts to degrade — unlike realistic-document benchmarks like LongBench v2, which blend retrieval with task-specific reasoning over genuine long texts. ## Run RULER on Your Model Benchgen lets you evaluate your own model against RULER's synthetic long-context suite at 128K tokens and track results across model versions. ## Related Models & Benchmarks - Top performers: [Granite 4.2 30B](/models/ibm/granite-4-2-30b) · [Granite 4.2 8B](/models/ibm/granite-4-2-8b) · [LFM3-1.7B](/models/liquid-ai/lfm3-1-7b) · [Granite 4.2 3B](/models/ibm/granite-4-2-3b) - Related benchmarks: [RULER 64K](/benchmarks/nvidia/ruler-64k) · [MRCRv2](/benchmarks/google-deepmind/mrcrv2) · [LongBench v2](/benchmarks/longbench-v2) ## Frequently Asked Questions

What is RULER?

RULER is a synthetic long-context benchmark introduced by Hsieh, Sun, Kriman, Acharya, Rekesh, Jia, Zhang & Ginsburg at NVIDIA (COLM 2024), expanding needle-in-a-haystack retrieval with multi-hop tracing and aggregation tasks across 13 task configurations and configurable sequence lengths.

What does a good RULER 128K score look like?

IBM's Granite 4.2 30B reports 81.38% as of August 2026, the highest currently tracked on Benchgen at this length; Liquid AI's LFM3-1.7B scores 65.56% despite being a much smaller 1.73B-parameter edge model.

Who created RULER?

RULER was created by Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg at NVIDIA; see the [original paper](https://arxiv.org/abs/2404.06654), published at COLM 2024.

Why does RULER have separate pages per context length?

RULER's central finding is that model accuracy degrades as sequence length increases, so a single blended score would hide that pattern; Benchgen splits RULER into per-length pages (64K, 128K) the same way it splits BrowseComp Long Context into 128K/256K variant pages, so scores stay comparable at a fixed length.

Is RULER saturated?

No — the original paper found that even models claiming 128K+ token context windows often show large accuracy drops well before that length, confirming RULER remains a low-saturation benchmark at 128K.

--- *Benchmark definition paraphrased from [Hsieh et al. 2024](https://arxiv.org/abs/2404.06654), COLM 2024. State-of-the-art scores sourced from published model cards/blogs and attributed inline. Last updated 2026-10-31.*

Rank	Model	Score
1	granite-4-2-30b	81.38
2	granite-4-2-8b	71.41
3	lfm3-1-7b	65.56
4	granite-4-2-3b	55.3

RULER 128K — Results

RULER 128K — Evaluations