MRCRv2 1M

MRCR v2 (8-needle, 1M)

MRCR (Multi-Round Coreference Resolution) comes from Google DeepMind's Michelangelo paper (arXiv 2409.12640). The model reads a long conversation in which a user asks for many pieces of writing (a format, a topic and a style each) and the assistant writes them. Eight of the requests are the same. At the end the model is asked to reproduce one of those eight, for example the fourth, and to start its answer with a random 12-character string.

This benchmark runs one context length of Google DeepMind's public data release v2.1 (eval_hub/mrcr_v2, Apache-2.0): 310 8-needle questions, all on the same conversation of about 1,048,384 tokens. This is the number reported as "MRCR v2 (8-needle) 1M pointwise". The MRCRv2 benchmark has the 4k to 128k average.

Scoring

Each answer is scored with Google DeepMind's reference metric: the answer must contain the question's random string, and what follows its last occurrence is compared with the right piece of writing (difflib SequenceMatcher, 0 to 1). A missing string scores 0.

• Score: the mean over all 310 questions, times 100.
• Coverage: the share of questions the model answered. Every prompt is about 1,048,384 tokens (Google DeepMind's tokenizer), so a model needs a context window of more than that, with room for the answer.

The model gets the conversation as one message, with no system prompt and no tools. Tools make the task much easier and the score not comparable.

Cost

A run sends about 325 million input tokens. Every question repeats the same conversation, so models with prompt caching cost far less than the list price. "Questions" on the run form asks fewer for a quick look.