Skip to main content
MRCRv2 is Google DeepMind’s Multi-Round Coreference Resolution benchmark (data release v2.1, 8 needles). The model reads a long conversation in which a user asked for many pieces of writing, eight of them alike, and has to reproduce the right one, for example the fourth poem about flamingos, starting with a random 12-character string. There are 484 questions in five context-length bins, from 4k to 128k tokens. Score is the mean over all of them, the number labs publish as “MRCR v2 (8-needle) 128k (average)”. Your model does not have to live on BenchGen to take it. If it answers on an OpenAI-compatible chat completions endpoint (OpenRouter, OpenAI, Together, Mistral, or your own vLLM server), the External API source runs MRCRv2 against it. This guide walks through one run: google/gemini-2.5-flash-lite on OpenRouter, one question per bin for a quick first look.

The whole walkthrough as a video

Before you start

  • A BenchGen account with at least 5 credits (a run cannot start below that).
  • An endpoint that serves POST .../chat/completions in the OpenAI format and is reachable from the public internet.
  • The model name your endpoint expects, for example google/gemini-2.5-flash-lite.
  • An API key, if your endpoint needs one.
  • A model with a context window above 128k tokens, to answer every question. MRCRv2 prompts reach about 131,000 tokens; a shorter window still works, its longest questions just count as unanswered.

How your endpoint is called

The benchmark never talks to your endpoint directly. For the length of the run, BenchGen registers your endpoint on its gateway and gives the benchmark container a key made for this run only. Every request goes through the gateway to your endpoint with your own key, which is why each call shows up as a trace on the run page.
Your API key is held by the gateway only while the run lasts and is removed when the run ends. It never reaches the benchmark container, is not saved with the run, and is not stored as a reusable model.

Run the benchmark

1

Open MRCRv2

Open MRCRv2 from the Environments page and select the Evaluate tab.
The Evaluate tab of the MRCRv2 benchmark

The Evaluate tab of MRCRv2

2

Choose External API

Click External API in the row of model sources.
3

Enter your endpoint

The URL may end in /v1 or in /v1/chat/completions; both work. Put the key in its own field, not in the URL.
The External API form with the endpoint URL, model name, masked key and display name filled in

The External API form filled in for Gemini 2.5 Flash Lite on OpenRouter

4

Check the connection

Click Use this endpoint. BenchGen sends one short request to your endpoint through the gateway. When it answers, the form shows Selected: with your display name and (the endpoint answered).
The form showing that the endpoint answered and is selected

The endpoint answered, and it is selected for the run

If the request fails, the form says why, in one line, with the endpoint’s own message. See Troubleshooting. For an endpoint that is slow to wake up, you can click Use anyway and start the run.
5

Set the run in Advanced (optional)

Expand Advanced: model environment & parameters. For a first look, type 1 in Questions per bin and leave everything else blank. Advanced settings below explains every field.
The Advanced panel with Questions per bin set to 1

Questions per bin set to 1 for a quick first run

6

Run the evaluation

Click Run Evaluation. The run page opens and follows the run through Deploying and Running to Completed, with the benchmark’s log underneath.
7

Read the results

When the run completes, Score Breakdown shows:
  • Score: the mean over the questions asked, 0 to 100. Each answer scores 0 to 1 by how closely the text after the random string matches the right piece of writing; an answer without the string scores 0.
  • 4k-8k to 64k-128k: the same mean within each context-length bin, so you can see where the model starts to lose track.
  • Coverage: the share of all 484 questions the model answered. A prompt longer than the model’s context window counts as unanswered and scores 0.
The completed run page with the score breakdown

A completed run with its score, bins and coverage

A run with Questions per bin set is an estimate over the questions it asked, not the full-set average labs publish. Its coverage stays a share of all 484, so a partial run is plain to see on the leaderboard.
8

Inspect every model call

Click Questions & traces on the run page. The table lists every request the run sent to your endpoint, oldest first, with its latency and cost. Click a row to open the full trace: the prompt, the answer, and the model’s reasoning when it returns any.
The traces table of the run with one trace opened beside it

Every model call of the run, and one of them opened

To keep traces, tick them and click Download in the bar that appears: you get one JSON file with each selected trace in full. Copy IDs copies their ids. Only you and BenchGen staff can see a run’s traces.

Advanced settings

Advanced: model environment & parameters holds two kinds of fields. Every field is optional: a blank one takes the benchmark’s default, shown greyed out inside it.

You provide these

The fields a benchmark declares for its runner. MRCRv2 declares two; other benchmarks declare their own. Each one reaches the benchmark as the environment variable shown next to its label.

Sampling parameters

These go to your endpoint with every request. MRCRv2’s defaults are the ones Google DeepMind’s own runner uses; change them and your score is no longer comparable with published numbers.

What it costs

The run page shows the compute amount under Compute once the run has ended, and Billing counts it under Benchmarks on the day it was charged. A full MRCRv2 run sends about 24.5 million input tokens to your endpoint; questions share their conversations, so a provider with prompt caching charges far less than its list price.

Run it again

The key is not kept after a run, so an External API run has no one-click rerun. To run the same endpoint again, start a new evaluation and enter the URL, model name and key once more.

Troubleshooting

Next steps

Benchmark results

Read the score breakdown, the detailed results and the files of a run.

Environment variables

How a benchmark’s run settings reach the model.

Longer contexts

MRCRv2 256K, 512K and 1M ask every question at one context length, for models with longer windows.

Model usage

Follow what your runs spend.
Last modified on October 23, 2026