Router Fidelity Benchmark

Router Fidelity Benchmark

46 tasks, English math/knowledge/reasoning questions drawn from the SAME distribution a BenchGen router head is trained on: MATH500, MMLU, MMLU-Pro, ARC-Challenge, GSM8K, and AIME2025, joined from benchgen/router-pilot (reward labels) and benchgen/router-pilot-tasks (prompts/gold answers).

Why this benchmark exists

A router head embeds each query with a frozen backbone and picks a pool member based on where that embedding falls relative to what it saw during training. Evaluate it on data from a wildly different distribution (a different language, a different domain) and the embedding lands somewhere the classifier never learned to discriminate — its pick becomes arbitrary, not a genuine routing decision, and it tends to collapse to a single agent. This benchmark's tasks are drawn from the exact same sources the router trained on, so a run here is a real fidelity check: does the routing decision hold up on the data it was actually built for?

Task

Each task's prompt already includes its own answer-format instruction (Answer: X for multiple choice, \boxed{} for a math expression, Answer: <number> for a numeric answer) — forwarded to the model verbatim.

Scoring

Deterministic, grader-aware per task (mcq_letter / numeric_match / math_expression) — no LLM-as-judge required. Reports overall accuracy plus breakdowns by difficulty (easy/medium/hard) and domain (math/knowledge/reasoning).

Routing report

If the evaluated model is a BenchGen router model, the results page shows which pool member (and the real underlying model, e.g. openai/gpt-oss-120b) answered each question, plus an overall routing distribution — empty for a non-router model. Click any question in the detailed results to expand its full text.

Cost KPI

Every task's real token usage and cost (in USD) is captured and shown, so a router model's actual cost can be compared directly against a single fixed model on the SAME leaderboard: same accuracy, lower total cost is the concrete case for routing. A router model's cost is real (the gateway knows each picked agent's own published price); a plain baseline model's cost is auto-detected at run start from OpenRouter's public model catalog — no config needed. Detailed results show cost + tokens per task, plus total cost, average cost per task, and total tokens as headline stats.

Env-var-first

Model connection is given via environment variables, not model.py. The platform injects your selected model's api_url / api_key / model_name into the container (RPE_* prefix, or the generic MODEL_* keys).

From the run screen's Advanced panel you can add:

• RPE_MAX_QUESTIONS — only run the first N tasks (for a quick test, e.g. 5).