> ## Documentation Index
> Fetch the complete documentation index at: https://benchgen.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# MRCRv2 with an External API model

> Run Google DeepMind's MRCR v2 long-context benchmark on a model behind your own OpenAI-compatible endpoint, tune the run in Advanced, and read the score and every model call.

**MRCRv2** is Google DeepMind's Multi-Round Coreference Resolution benchmark (data release v2.1,
8 needles). The model reads a long conversation in which a user asked for many pieces of writing,
eight of them alike, and has to reproduce the right one, for example the fourth poem about
flamingos, starting with a random 12-character string. There are 484 questions in five
context-length bins, from 4k to 128k tokens. **Score** is the mean over all of them, the number labs
publish as "MRCR v2 (8-needle) 128k (average)".

Your model does not have to live on BenchGen to take it. If it answers on an OpenAI-compatible chat
completions endpoint (OpenRouter, OpenAI, Together, Mistral, or your own vLLM server), the
**External API** source runs MRCRv2 against it. This guide walks through one run:
`google/gemini-2.5-flash-lite` on OpenRouter, one question per bin for a quick first look.

<video controls src="https://mintcdn.com/benchgen-8fc81371/_sV37YBfErxDwhvI/images/guides/benchmarks/mrcrv2-external-model.mp4?fit=max&auto=format&n=_sV37YBfErxDwhvI&q=85&s=d1a7223e621e4f018ed0e46b5b620f2e" />

The whole walkthrough as a video

## Before you start

* A BenchGen account with at least **5 credits** (a run cannot start below that).
* An endpoint that serves `POST .../chat/completions` in the OpenAI format and is reachable from the
  public internet.
* The model name your endpoint expects, for example `google/gemini-2.5-flash-lite`.
* An API key, if your endpoint needs one.
* A model with a context window above 128k tokens, to answer every question. MRCRv2 prompts reach
  about 131,000 tokens; a shorter window still works, its longest questions just count as
  unanswered.

## How your endpoint is called

The benchmark never talks to your endpoint directly. For the length of the run, BenchGen registers
your endpoint on its gateway and gives the benchmark container a key made for this run only. Every
request goes through the gateway to your endpoint with your own key, which is why each call shows
up as a trace on the run page.

> [!NOTE]
>
> Your API key is held by the gateway only while the run lasts and is removed when the run ends. It
> never reaches the benchmark container, is not saved with the run, and is not stored as a reusable
> model.

## Run the benchmark

**1. Open MRCRv2**

Open **MRCRv2** from the **Environments** page and select the **Evaluate** tab.

![The Evaluate tab of the MRCRv2 benchmark](https://mintcdn.com/benchgen-8fc81371/_sV37YBfErxDwhvI/images/guides/benchmarks/01-evaluate-tab.png?fit=max\&auto=format\&n=_sV37YBfErxDwhvI\&q=85\&s=2766c9ade5350c06f77a3ce684b73fca)

The Evaluate tab of MRCRv2

**2. Choose External API**

Click **External API** in the row of model sources.

**3. Enter your endpoint**

| Field | Required | Example |
| - | - | - |
| **Endpoint URL** | Yes | `https://openrouter.ai/api/v1/chat/completions` |
| **Model name** | Yes | `google/gemini-2.5-flash-lite` |
| **API key / token** | If your endpoint needs one | Your OpenRouter key |
| **Display name** | No | `Gemini 2.5 Flash Lite (OpenRouter)`, the name the run is listed under |

The URL may end in `/v1` or in `/v1/chat/completions`; both work. Put the key in its own field,
not in the URL.

![The External API form with the endpoint URL, model name, masked key and display name filled in](https://mintcdn.com/benchgen-8fc81371/_sV37YBfErxDwhvI/images/guides/benchmarks/02-endpoint-form.png?fit=max\&auto=format\&n=_sV37YBfErxDwhvI\&q=85\&s=0a70be45eac9943e62093049063f0351)

The External API form filled in for Gemini 2.5 Flash Lite on OpenRouter

**4. Check the connection**

Click **Use this endpoint**. BenchGen sends one short request to your endpoint through the
gateway. When it answers, the form shows **Selected:** with your display name and **(the
endpoint answered)**.

![The form showing that the endpoint answered and is selected](https://mintcdn.com/benchgen-8fc81371/_sV37YBfErxDwhvI/images/guides/benchmarks/03-endpoint-checked.png?fit=max\&auto=format\&n=_sV37YBfErxDwhvI\&q=85\&s=adef94cea87154bd7bef456dec16b355)

The endpoint answered, and it is selected for the run

If the request fails, the form says why, in one line, with the endpoint's own message. See
[Troubleshooting](#troubleshooting). For an endpoint that is slow to wake up, you can click
**Use anyway** and start the run.

**5. Set the run in Advanced (optional)**

Expand **Advanced: model environment & parameters**. For a first look, type `1` in **Questions
per bin** and leave everything else blank. [Advanced settings](#advanced-settings) below
explains every field.

![The Advanced panel with Questions per bin set to 1](https://mintcdn.com/benchgen-8fc81371/_sV37YBfErxDwhvI/images/guides/benchmarks/04-advanced.png?fit=max\&auto=format\&n=_sV37YBfErxDwhvI\&q=85\&s=5f570f6c97f83098cfe030cf4ac63edf)

Questions per bin set to 1 for a quick first run

**6. Run the evaluation**

Click **Run Evaluation**. The run page opens and follows the run through **Deploying** and
**Running** to **Completed**, with the benchmark's log underneath.

**7. Read the results**

When the run completes, **Score Breakdown** shows:

* **Score**: the mean over the questions asked, 0 to 100. Each answer scores 0 to 1 by how
  closely the text after the random string matches the right piece of writing; an answer
  without the string scores 0.
* **4k-8k** to **64k-128k**: the same mean within each context-length bin, so you can see where
  the model starts to lose track.
* **Coverage**: the share of all 484 questions the model answered. A prompt longer than the
  model's context window counts as unanswered and scores 0.

![The completed run page with the score breakdown](https://mintcdn.com/benchgen-8fc81371/_sV37YBfErxDwhvI/images/guides/benchmarks/05-results.png?fit=max\&auto=format\&n=_sV37YBfErxDwhvI\&q=85\&s=5089fec900d49ba4c224602a3a413d1d)

A completed run with its score, bins and coverage

> [!TIP]
>
> A run with **Questions per bin** set is an estimate over the questions it asked, not the
> full-set average labs publish. Its coverage stays a share of all 484, so a partial run is plain
> to see on the leaderboard.

**8. Inspect every model call**

Click **Questions & traces** on the run page. The table lists every request the run sent to
your endpoint, oldest first, with its latency and cost. Click a row to open the full trace: the
prompt, the answer, and the model's reasoning when it returns any.

![The traces table of the run with one trace opened beside it](https://mintcdn.com/benchgen-8fc81371/_sV37YBfErxDwhvI/images/guides/benchmarks/06-traces.png?fit=max\&auto=format\&n=_sV37YBfErxDwhvI\&q=85\&s=3ccaf9b9898b81f7316f83a45cf83676)

Every model call of the run, and one of them opened

To keep traces, tick them and click **Download** in the bar that appears: you get one JSON file
with each selected trace in full. **Copy IDs** copies their ids. Only you and BenchGen staff
can see a run's traces.

## Advanced settings

**Advanced: model environment & parameters** holds two kinds of fields. Every field is optional:
a blank one takes the benchmark's default, shown greyed out inside it.

### You provide these

The fields a benchmark declares for its runner. MRCRv2 declares two; other benchmarks declare
their own. Each one reaches the benchmark as the environment variable shown next to its label.

| Field | Variable | MRCRv2 default | What it does |
| - | - | - | - |
| **Questions per bin** | `MRCR_PER_BIN` | Blank: all 484 | Asks N questions from each of the five bins, spread evenly over the bin and the same ones every run. `1` asks 5 questions, `10` asks 50. |
| **Parallel questions (1-16)** | `MRCR_WORKERS` | 8 | How many questions run at the same time. Lower it if your provider answers with rate-limit errors; raise it for a faster full run on an endpoint that can take it. |

### Sampling parameters

These go to your endpoint with every request. MRCRv2's defaults are the ones Google DeepMind's own
runner uses; change them and your score is no longer comparable with published numbers.

| Field | MRCRv2 default | What it does |
| - | - | - |
| **Temperature** | 1.0 | Sampling temperature. |
| **Max tokens** | 32,768 | Room for the answer. Answers are short, the rest is for models that reason first. If a prompt plus this room does not fit the model's context window, the question is asked again with 8,192, then with 1,024. |
| **Top P** | 1.0 | Nucleus sampling. |
| **Timeout (s)** | 900 | How long one question may take before it is retried; long prompts on a busy endpoint need the time. |
| **Reasoning** | Your model's default | `none`, `low`, `medium` or `high`, sent as `reasoning_effort` to models that support it. More reasoning usually scores higher and costs more tokens. |

## What it costs

| | Who charges | Price |
| - | - | - |
| Model calls | Your provider, on your own key | Your provider's price. BenchGen adds nothing. |
| Benchmark compute | BenchGen, from your credits | \*\*$0.005 per minute** the run takes ($0.30 an hour) |

The run page shows the compute amount under **Compute** once the run has ended, and **Billing**
counts it under **Benchmarks** on the day it was charged. A full MRCRv2 run sends about 24.5 million
input tokens to your endpoint; questions share their conversations, so a provider with prompt
caching charges far less than its list price.

## Run it again

The key is not kept after a run, so an External API run has no one-click rerun. To run the same
endpoint again, start a new evaluation and enter the URL, model name and key once more.

## Troubleshooting

| What you see | What to do |
| - | - |
| **The endpoint refused the API key.** | Check the key, and that it belongs to the provider in the URL. |
| **Not found: check the endpoint URL and the model name.** | The URL path or the model name is wrong. Many providers answer 404 for an unknown model. |
| **The endpoint rejected the request: check the model name.** | Use the model name exactly as your provider lists it. |
| **The endpoint is rate limiting or out of quota.** | Wait, top up with your provider, or lower **Parallel questions**. |
| **No answer within 15 seconds.** | The endpoint may be starting up. Try again, or click **Use anyway**. |
| **... is not a public host name** or **... resolves to a private or internal address** | BenchGen only calls endpoints on the public internet. A server on `localhost` or a private network needs a public URL first. |
| **The endpoint URL must not have a query string** | Remove everything from `?` on. |
| Low **Coverage** | The model's context window is shorter than the longest prompts (about 131,000 tokens). |

## Next steps

> [card] [Benchmark results](https://benchgen.com/docs/eval/read-results.md)
>
> Read the score breakdown, the detailed results and the files of a run.

> [card] [Environment variables](https://benchgen.com/docs/eval/environment-variables.md)
>
> How a benchmark's run settings reach the model.

> [card] [Longer contexts](https://benchgen.com/benchmarks)
>
> **MRCRv2 256K**, **512K** and **1M** ask every question at one context length, for models with
> longer windows.

> [card] [Model usage](https://benchgen.com/docs/eval/monitor-model-usage.md)
>
> Follow what your runs spend.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.