BenchGen Router Lite

Not every query needs the same model. A frontier-class model is expensive and often overkill for an easy question, and a cheap model quietly fails on a hard one. The usual answer is to pick one model for everything, or to hand-write routing rules. BenchGen Router Lite takes a different approach, similar in spirit to Sakana AI’s work on model routing: train a small, dedicated router that learns, from measured evidence rather than guesses, which model in a pool is the right one for a given query. The router itself never generates an answer. It looks at a query, picks the best pool member for it, and that pool member’s real reply is what comes back. This guide tells the complete story of how BenchGen Router Lite was built: the dataset it learned from, how it was trained, and how it was benchmarked.

Diagram of one query moving through BenchGen Router Lite: the frozen backbone embeds it, the router head picks one pool member, here open_mid, and only that model's real reply is returned — One query moving through BenchGen Router Lite: the frozen backbone embeds it, the head picks one pool member (here open_mid), and only that model's real reply comes back. No text is ever generated by the head itself.

1. Build the dataset

A router has nothing to learn from until you can prove that different models are actually better at different things. That proof is a reward matrix: for every task, how well each candidate model actually scored on it, measured rather than assumed. BenchGen Router Lite trains on two paired datasets published from a single pilot collection run.

Dataset	Contents	Why it is separate
Rewards dataset	`task_id`, one reward column per pool member, tie and empty/error flags	Published without prompt text, since redistribution rights differ per upstream source
Tasks dataset	`task_id`, `prompt`, gold answer, grader, domain, difficulty, split	Supplies the prompt text so a reward row can be embedded and joined at training time

The join key is task_id. Training the router head means loading both datasets and matching rows on that field before anything gets embedded.

The reward dataset

Every model was reached through OpenRouter, so the pool could mix closed-style and open-weight models behind one API and one billing account, and every candidate went through the same pre-flight check before being trusted with real collection: 20 live calls against a trivial prompt, checking that the model returned a valid completion, respected the max_tokens cap instead of silently truncating, reported usable token counts, and kept its empty-response rate under 5%. The five models that passed pre-flight split into two roles inside the pool:

Role	Agent	Model	Mode
Frontier	`frontier_a`	`openai/gpt-oss-120b`	Reasoning
Frontier	`frontier_b`	`deepseek/deepseek-v4-flash-0731`	Reasoning
Frontier	`frontier_c`	`google/gemma-3-27b-it`	Direct
Mid-tier	`open_mid`	`mistralai/mistral-nemo`	Direct
Cheap reasoning	`open_cheap_reasoning`	`inclusionai/ling-3.0-flash`	Reasoning

The three Frontier slots are the pool’s strong anchors, the models a naive “always use one model” strategy would pick, standing in for closed, proprietary frontier APIs on a budget that favored strong open-weight models available on OpenRouter. The Mid-tier slot is the deliberately weakest agent in the pool, included specifically to see if it is ever uniquely best on cheap questions, and the cheap reasoning slot is the cheapest reasoning-capable model in the catalogue, from a different vendor family so its failures do not correlate with the frontier slots. This composition mirrors the coordinator design in Sakana AI’s Trinity paper (arXiv:2512.04695): a router only has something to learn if no single agent dominates everywhere. A pool of near-identical strong models gives a router nothing to route between; a pool with a genuinely weaker, cheaper option gives it a real decision to make. Every agent was called under identical settings, so a score difference reflects the model and not the prompt conditions:

Setting	Value
`max_tokens`	4,096
`temperature`	0.1
`top_p`	1.0
Reasoning effort	Minimal
Repetitions	3 per task, per agent
Timeout	180 seconds, up to 3 retries

Answers are extracted in order: a tagged answer, boxed LaTeX, a labelled answer, then a last-number fallback that strips markup first. An empty response is never scored correct, and empty responses are tracked separately from provider errors so a broken transport is never mistaken for a weak model. The pilot scored 60 tasks across 8 source datasets. One source, rlpr, was collected but excluded from the public release, since its answers are free-form prose and tables and the string-match grader used here scores genuinely correct RLPR answers as wrong. A label that cannot be trusted is worse than no label, so those 14 tasks were dropped before publishing. That leaves 46 tasks x 5 agents x 3 repetitions = 690 graded calls, published as the rewards dataset for a total collection cost of $0.0826 across the full pilot, with zero call errors. A reward row looks like this:

{
  "task_id": "aime2025_i/test/000002",
  "agent_order": ["frontier_a", "frontier_b", "frontier_c", "open_mid", "open_cheap_reasoning"],
  "mean_reward": [1.0, 1.0, 0.6666666666666666, 0.0, 1.0],
  "best_agents": ["frontier_a", "frontier_b", "open_cheap_reasoning"],
  "is_tie": true
}

mean_reward[i] is agent agent_order[i]’s accuracy over 3 attempts. Ties are kept explicit in best_agents rather than collapsed to a single winner, because on easy questions several agents are genuinely equal, and pretending otherwise teaches a router a coin flip.

Nothing published contains question text or raw model completions. Upstream licences differ per source and provider terms on republishing outputs were not cleared for every agent, so the reward signal (correctness, ties, empties, errors) is published without either.

The task dataset

The task pool is a normalized set of prompts drawn from eight public benchmark sources, balanced across domains and split with a fixed seed so results are reproducible.

Source	Tasks	Domain	Licence
`math500`	233	Math	MIT
`rlpr`	250	Reasoning	Apache-2.0
`mmlu_pro`	186	Knowledge	Unverified at collection time
`mmlu`	184	Knowledge	MIT
`arc_challenge`	120	Reasoning	CC-BY-SA-4.0
`gsm8k`	109	Math	MIT
`aime2025_i`	15	Math	MIT
`aime2025_ii`	13	Math	MIT
Total	1,110

Domain, difficulty, and split are balanced deliberately, not incidentally:

Domain

Knowledge: 370 (33.3%)
Math: 370 (33.3%)
Reasoning: 370 (33.3%)

Difficulty

Medium: 504 (45.4%)
Hard: 427 (38.5%)
Easy: 179 (16.1%)

Split

Train: 648 (58.4%)
Test: 245 (22.1%)
Validation: 217 (19.5%)

All 1,110 tasks are published as the paired tasks dataset. Only the 46 tasks described above currently carry reward labels; the rest of the pool exists so future collection rounds, on a wider agent pool or a bigger budget, can reuse the same balanced task set instead of rebuilding it.

What the pilot shows before any router gets trained

Every dataset ships with baseline numbers, because a routing dataset that only reports the winner’s score invites the exact mistake this project exists to avoid: training a router on a pool where one model already wins everywhere.

Policy	Mean reward
`frontier_a`	0.6667
`open_cheap_reasoning`	0.6667
`frontier_b`	0.6410
`frontier_c`	0.5641
`open_mid`	0.2308
Uniform random pick	0.5538
Best single fixed agent	0.6667
Per-question oracle (upper bound)	0.7692

The gap between the best fixed agent and the oracle, 0.1026, is the routing headroom: the most a trained router could possibly gain over always calling frontier_a. That is the bar the trained router is measured against in Benchmark below.

2. Train the router head

BenchGen Router Lite does not fine-tune a model, and it does not train a LoRA adapter on top of one. What gets trained is a router head: a small classifier, a few thousand parameters, that sits on top of a frozen backbone and learns which pool member to send each query to. The backbone never updates. Only the head does.

	Full fine-tune / LoRA	Router head
What changes	Model weights (or an adapter)	A small classifier on top of a frozen backbone
What it learns	How to answer	Which pool member should answer
Training method	Gradient descent over many steps	Separable CMA-ES, a gradient-free evolutionary search
Output at inference	Generated text	A routing decision; the picked pool member’s real reply is what gets returned

The head is trained with separable CMA-ES against the static reward matrix from the dataset above, not by calling any pool model during training. Every (task, agent) reward is already in the dataset, so a full training run makes zero API calls and costs nothing beyond the embedding pass.

1. Start a new training run and pick Head

In the Train tab, click + New Training, then pick Head on the training method toggle (“Learn from past agent scores”). Point Base Model at the frozen backbone (this run used Qwen/Qwen3-1.7B) and Dataset at the rewards dataset, benchgen/router-pilot.

Training method set to Head, with Qwen3-1.7B as the base model and router-pilot as the dataset — The New Training form with Head selected, Qwen/Qwen3-1.7B as the base model, and router-pilot as the dataset

2. Fill in the head-specific fields

Add the Tasks dataset, benchgen/router-pilot-tasks, so prompts can be joined and embedded. Set Held-out split value to test, the split the dataset’s own reported baselines use, and leave Head hidden dimension at 0 for a pure linear head. This run used the defaults for the rest: sigma0 0.5, 60 generations, automatic population size, and seed 42, with the CUDA embedding device.

The Head-specific fields: tasks dataset, held-out split, head hidden dimension, and CMA-ES hyperparameters — The Head-specific fields: tasks dataset, held-out split value, head hidden dimension, and the sep-CMA-ES hyperparameters (sigma0, generations, population, seed)

3. Submit and watch it train

Click Start Training. The logs confirm exactly what is loading before anything else happens:

=== Router head training: head-Qwen3-1.7B-e99b449fde ===
Loading rewards dataset benchgen/router-pilot (split=train)...
  46 reward rows loaded.
Loading tasks dataset benchgen/router-pilot-tasks (split=train) ...
  1110 tasks with prompt text loaded.
Pool (5 agents): ['frontier_a', 'frontier_b', 'frontier_c', 'open_mid', 'open_cheap_reasoning']

Training logs showing the reward dataset, tasks dataset, and five-agent pool loading — Training logs right after submit: the rewards dataset, the tasks dataset, and the five-agent pool all loading before embedding starts

4. CMA-ES evolves the head

The backbone embeds each prompt once, splits into 35 train rows and 11 test rows, and then a 10,245-parameter head is evolved against the training rows:

Train rows: 35, test rows: 11, embedding dim: 2048
sep-CMA-ES: 10245 head parameters, sigma0=0.5, generations=60, popsize=auto
  gen 0: best_reward_so_far=0.9333
  gen 1: best_reward_so_far=0.9429
  gen 2: best_reward_so_far=0.9429
  gen 3: best_reward_so_far=0.9429

Training progress and logs showing sep-CMA-ES generations and the best training reward so far — Training Progress at 100% alongside the raw logs: 35 train rows, 11 test rows, a 10,245-parameter head, and the best training reward climbing across generations

This whole run, embedding included, finished in 6 seconds on a single auto-selected GPU. That best_reward_so_far number is the head’s fit to the 35 training rows as CMA-ES optimizes, not a held-out score. What the trained head actually generalizes to is what Benchmark below measures.

5. Register the head

A completed job’s Use In Benchmarks panel offers Register as Model. Click it, and the head becomes selectable anywhere BenchGen lets you pick a model, carrying a Head badge and a stable name (head-Qwen3-1.7B-e99b449fde for this run) so it is never confused with a regular fine-tuned model. There is no merge or export step, since a router head has no adapter weights to fold into a base model.

A completed and registered router head, with its head-Qwen3-1.7B-e99b449fde identifier — The completed job, registered and ready: head-Qwen3-1.7B-e99b449fde, selectable anywhere BenchGen lets you pick a model

3. Benchmark

A router head’s own output is a routing decision, not an answer. Running it through a benchmark scores the router and its pool together, not the head in isolation, since every response a user sees comes from whichever pool member the head picked. BenchGen Router Lite was evaluated against TR Benchmark (TR, Karma Zorluk), an existing BenchGen environment picked deliberately because it is nothing like the training data: 70 Turkish multiple-choice questions, 50 easy and medium general-knowledge questions plus 20 genuinely hard ones (multi-step math, logic, obscure facts), each with four options and a single correct letter. Scoring is deterministic exact-match on the letter, with no LLM-as-judge involved, and because the model under test is a router, the results table also records which pool member answered each question.

The TR Benchmark overview page describing the 70-question Turkish benchmark and its scoring rules — The TR Benchmark (TR, Karma Zorluk) overview: 70 Turkish multiple-choice questions, deterministic scoring, with a sample question shown

This is an out-of-distribution test on purpose. The reward dataset was built from English math, knowledge, and reasoning sources; this benchmark is Turkish general knowledge and logic. Evaluating here checks whether the router’s routing decision holds up outside the exact domain it learned from, not just inside it.

Running the evaluation

From the benchmark’s Evaluate tab, the registered head shows up under Platform Models alongside every other trained model, carrying its Head badge. Selecting it auto-fills the sampling parameters (temperature 0.7, max tokens 150, top P 1.0, 30 second timeout) from the model, and Run Evaluation starts the same round trip described above: each question gets embedded, the head picks a pool member, and that pool member answers for real.

Selecting the registered benchgen-router-lite head under Platform Models, with sampling parameters auto-filled — Selecting benchgen-router-lite under Platform Models, with sampling parameters auto-filled from the registered head

The evaluation running live, streaming step-by-step progress through the 70 questions — The evaluation running live, streaming through the 70 questions one step at a time

Watching the questions arrive at the pool

Because the router never generates text itself, every question it routes turns into a real API call to the selected pool member. Those calls show up in the OpenRouter dashboard as the run proceeds, which is the clearest way to confirm the routing is genuinely happening rather than being simulated. The generations log lists each call with the model that received it, the provider that served it, token counts, and cost. The three pool members that were reachable during this run appear here by name: gpt-oss-120b, DeepSeek V4 Flash 0731, and Ling-3.0-flash.

OpenRouter logs listing generations by model, provider, token counts, and cost — The OpenRouter generations log during the benchmark run, one row per routed question with model, provider, tokens, and cost

Opening a single call shows the exact prompt the pool member received: the benchmark question, its four options, and the instruction to answer with a single letter. This is the benchmark’s own formatting, passed through untouched by the router.

OpenRouter prompt view showing a multiple choice benchmark question and the answer format instruction — One routed question as the pool member received it: the benchmark prompt, its options, and the single-letter answer instruction

The generation details confirm which key issued the call, benchgen-router-lite, along with the routing and generation timings and the prompt and completion token counts for that single question.

OpenRouter generation details showing the benchgen-router-lite API key, provider routing timings, and token counts — Generation details for one routed call, issued by the benchgen-router-lite key, with routing timings and token counts

Results

Overall score: 72.9% (51 of 70 correct, 1 empty response), broken down by difficulty:

Difficulty	Score
Easy	78.0% (39/50)
Medium	84.6% (11/13)
Hard	14.3% (1/7)

The routing distribution tells the more interesting story: 69 of 70 questions were routed to open_mid (mistralai/mistral-nemo). The one remaining question came back with no routing decision recorded at all, which is also the run’s single empty response.

Score breakdown and detailed per-question results, showing the 72.9% overall score and the routing distribution — The completed run: 72.9% overall, with the per-question table showing which pool member answered each question and whether it was correct

Explore it on BenchGen

The dataset and the trained head both live on the BenchGen platform, where you can inspect their details, schema, and files. Each page also links onward to the underlying repository if you want to download the raw artifacts.

Router rewards dataset

46 tasks, 5 agents, 3 repetitions. The reward matrix the router head trains against, paired with the 1,110-task prompt pool it joins on task_id.

The trained head-Qwen3-1.7B-e99b449fde head: a 10,245-parameter weight vector, its manifest, and a reference implementation of the embed and pick step.

​1. Build the dataset

​The reward dataset

​The task dataset

Domain

Difficulty

Split

​What the pilot shows before any router gets trained

​2. Train the router head

​1. Start a new training run and pick Head

​2. Fill in the head-specific fields

​3. Submit and watch it train

​4. CMA-ES evolves the head

​5. Register the head

​3. Benchmark

​Running the evaluation

​Watching the questions arrive at the pool

​Results

​Explore it on BenchGen

Router rewards dataset