> ## Documentation Index > Fetch the complete documentation index at: https://benchgen.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # BenchGen Router Lite > The full story of BenchGen Router Lite: a lightweight routing model that learns which language model should answer each query, instead of sending every query to the same one. Not every query needs the same model. A frontier-class model is expensive and often overkill for an easy question, and a cheap model quietly fails on a hard one. The usual answer is to pick one model for everything, or to hand-write routing rules. **BenchGen Router Lite** takes a different approach, similar in spirit to Sakana AI's work on model routing: train a small, dedicated router that learns, from measured evidence rather than guesses, which model in a pool is the right one for a given query. The router itself never generates an answer. It looks at a query, picks the best pool member for it, and that pool member's real reply is what comes back. This guide tells the complete story of how BenchGen Router Lite was built: the dataset it learned from, how it was trained, and how it was benchmarked. Diagram of one query moving through BenchGen Router Lite: the frozen backbone embeds it, the router head picks one pool member, here open_mid, and only that model's real reply is returned *** ## 1. Build the dataset A router has nothing to learn from until you can prove that different models are actually better at different things. That proof is a **reward matrix**: for every task, how well each candidate model actually scored on it, measured rather than assumed. BenchGen Router Lite trains on two paired datasets published from a single pilot collection run. | Dataset | Contents | Why it is separate | | --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | | **[Rewards dataset](https://benchgen.com/datasets/benchgen/benchgen-router-rewards-dataset)** | `task_id`, one reward column per pool member, tie and empty/error flags | Published without prompt text, since redistribution rights differ per upstream source | | **Tasks dataset** | `task_id`, `prompt`, gold answer, grader, domain, difficulty, split | Supplies the prompt text so a reward row can be embedded and joined at training time | The join key is `task_id`. Training the router head means loading both datasets and matching rows on that field before anything gets embedded. ### The reward dataset Every model was reached through [OpenRouter](https://openrouter.ai), so the pool could mix closed-style and open-weight models behind one API and one billing account, and every candidate went through the same pre-flight check before being trusted with real collection: 20 live calls against a trivial prompt, checking that the model returned a valid completion, respected the `max_tokens` cap instead of silently truncating, reported usable token counts, and kept its empty-response rate under 5%. The five models that passed pre-flight split into two roles inside the pool: | Role | Agent | Model | Mode | | ------------------- | ---------------------- | --------------------------------- | --------- | | **Frontier** | `frontier_a` | `openai/gpt-oss-120b` | Reasoning | | **Frontier** | `frontier_b` | `deepseek/deepseek-v4-flash-0731` | Reasoning | | **Frontier** | `frontier_c` | `google/gemma-3-27b-it` | Direct | | **Mid-tier** | `open_mid` | `mistralai/mistral-nemo` | Direct | | **Cheap reasoning** | `open_cheap_reasoning` | `inclusionai/ling-3.0-flash` | Reasoning | The three **Frontier** slots are the pool's strong anchors, the models a naive "always use one model" strategy would pick, standing in for closed, proprietary frontier APIs on a budget that favored strong open-weight models available on OpenRouter. The **Mid-tier** slot is the deliberately weakest agent in the pool, included specifically to see if it is ever uniquely best on cheap questions, and the **cheap reasoning** slot is the cheapest reasoning-capable model in the catalogue, from a different vendor family so its failures do not correlate with the frontier slots. This composition mirrors the coordinator design in Sakana AI's Trinity paper ([arXiv:2512.04695](https://arxiv.org/abs/2512.04695)): a router only has something to learn if no single agent dominates everywhere. A pool of near-identical strong models gives a router nothing to route between; a pool with a genuinely weaker, cheaper option gives it a real decision to make. Every agent was called under identical settings, so a score difference reflects the model and not the prompt conditions: | Setting | Value | | ---------------- | ---------------------------- | | `max_tokens` | 4,096 | | `temperature` | 0.1 | | `top_p` | 1.0 | | Reasoning effort | Minimal | | Repetitions | 3 per task, per agent | | Timeout | 180 seconds, up to 3 retries | Answers are extracted in order: a tagged answer, boxed LaTeX, a labelled answer, then a last-number fallback that strips markup first. An empty response is never scored correct, and empty responses are tracked separately from provider errors so a broken transport is never mistaken for a weak model. The pilot scored 60 tasks across 8 source datasets. One source, `rlpr`, was collected but excluded from the public release, since its answers are free-form prose and tables and the string-match grader used here scores genuinely correct RLPR answers as wrong. A label that cannot be trusted is worse than no label, so those 14 tasks were dropped before publishing. That leaves **46 tasks x 5 agents x 3 repetitions = 690 graded calls**, published as the [rewards dataset](https://benchgen.com/datasets/benchgen/benchgen-router-rewards-dataset) for a total collection cost of **\$0.0826** across the full pilot, with zero call errors. A reward row looks like this: ```json theme={null} { "task_id": "aime2025_i/test/000002", "agent_order": ["frontier_a", "frontier_b", "frontier_c", "open_mid", "open_cheap_reasoning"], "mean_reward": [1.0, 1.0, 0.6666666666666666, 0.0, 1.0], "best_agents": ["frontier_a", "frontier_b", "open_cheap_reasoning"], "is_tie": true } ``` `mean_reward[i]` is agent `agent_order[i]`'s accuracy over 3 attempts. Ties are kept explicit in `best_agents` rather than collapsed to a single winner, because on easy questions several agents are genuinely equal, and pretending otherwise teaches a router a coin flip. Nothing published contains question text or raw model completions. Upstream licences differ per source and provider terms on republishing outputs were not cleared for every agent, so the reward signal (correctness, ties, empties, errors) is published without either. ### The task dataset The task pool is a normalized set of prompts drawn from eight public benchmark sources, balanced across domains and split with a fixed seed so results are reproducible. | Source | Tasks | Domain | Licence | | --------------- | --------: | --------- | ----------------------------- | | `math500` | 233 | Math | MIT | | `rlpr` | 250 | Reasoning | Apache-2.0 | | `mmlu_pro` | 186 | Knowledge | Unverified at collection time | | `mmlu` | 184 | Knowledge | MIT | | `arc_challenge` | 120 | Reasoning | CC-BY-SA-4.0 | | `gsm8k` | 109 | Math | MIT | | `aime2025_i` | 15 | Math | MIT | | `aime2025_ii` | 13 | Math | MIT | | **Total** | **1,110** | | | Domain, difficulty, and split are balanced deliberately, not incidentally: Knowledge: 370 (33.3%)\ Math: 370 (33.3%)\ Reasoning: 370 (33.3%) Medium: 504 (45.4%)\ Hard: 427 (38.5%)\ Easy: 179 (16.1%) Train: 648 (58.4%)\ Test: 245 (22.1%)\ Validation: 217 (19.5%) All 1,110 tasks are published as the paired tasks dataset. Only the 46 tasks described above currently carry reward labels; the rest of the pool exists so future collection rounds, on a wider agent pool or a bigger budget, can reuse the same balanced task set instead of rebuilding it. ### What the pilot shows before any router gets trained Every dataset ships with baseline numbers, because a routing dataset that only reports the winner's score invites the exact mistake this project exists to avoid: training a router on a pool where one model already wins everywhere. | Policy | Mean reward | | ------------------------------------- | ----------: | | `frontier_a` | 0.6667 | | `open_cheap_reasoning` | 0.6667 | | `frontier_b` | 0.6410 | | `frontier_c` | 0.5641 | | `open_mid` | 0.2308 | | Uniform random pick | 0.5538 | | **Best single fixed agent** | **0.6667** | | **Per-question oracle (upper bound)** | **0.7692** | The gap between the best fixed agent and the oracle, **0.1026**, is the routing headroom: the most a trained router could possibly gain over always calling `frontier_a`. That is the bar the trained router is measured against in [Benchmark](#3-benchmark) below. *** ## 2. Train the router head BenchGen Router Lite does not fine-tune a model, and it does not train a LoRA adapter on top of one. What gets trained is a **router head**: a small classifier, a few thousand parameters, that sits on top of a frozen backbone and learns which pool member to send each query to. The backbone never updates. Only the head does. | | Full fine-tune / LoRA | Router head | | ------------------- | -------------------------------- | ----------------------------------------------------------------------------- | | What changes | Model weights (or an adapter) | A small classifier on top of a frozen backbone | | What it learns | How to answer | Which pool member should answer | | Training method | Gradient descent over many steps | Separable CMA-ES, a gradient-free evolutionary search | | Output at inference | Generated text | A routing decision; the picked pool member's real reply is what gets returned | The head is trained with **separable CMA-ES** against the static reward matrix from the dataset above, not by calling any pool model during training. Every `(task, agent)` reward is already in the dataset, so a full training run makes zero API calls and costs nothing beyond the embedding pass. ### 1. Start a new training run and pick Head In the **Train** tab, click **+ New Training**, then pick **Head** on the training method toggle ("Learn from past agent scores"). Point **Base Model** at the frozen backbone (this run used `Qwen/Qwen3-1.7B`) and **Dataset** at the rewards dataset, `benchgen/router-pilot`. Training method set to Head, with Qwen3-1.7B as the base model and router-pilot as the dataset ### 2. Fill in the head-specific fields Add the **Tasks dataset**, `benchgen/router-pilot-tasks`, so prompts can be joined and embedded. Set **Held-out split value** to `test`, the split the dataset's own reported baselines use, and leave **Head hidden dimension** at `0` for a pure linear head. This run used the defaults for the rest: `sigma0` 0.5, 60 generations, automatic population size, and seed 42, with the **CUDA** embedding device. The Head-specific fields: tasks dataset, held-out split, head hidden dimension, and CMA-ES hyperparameters ### 3. Submit and watch it train Click **Start Training**. The logs confirm exactly what is loading before anything else happens: ```text theme={null} === Router head training: head-Qwen3-1.7B-e99b449fde === Loading rewards dataset benchgen/router-pilot (split=train)... 46 reward rows loaded. Loading tasks dataset benchgen/router-pilot-tasks (split=train) ... 1110 tasks with prompt text loaded. Pool (5 agents): ['frontier_a', 'frontier_b', 'frontier_c', 'open_mid', 'open_cheap_reasoning'] ``` Training logs showing the reward dataset, tasks dataset, and five-agent pool loading ### 4. CMA-ES evolves the head The backbone embeds each prompt once, splits into 35 train rows and 11 test rows, and then a 10,245-parameter head is evolved against the training rows: ```text theme={null} Train rows: 35, test rows: 11, embedding dim: 2048 sep-CMA-ES: 10245 head parameters, sigma0=0.5, generations=60, popsize=auto gen 0: best_reward_so_far=0.9333 gen 1: best_reward_so_far=0.9429 gen 2: best_reward_so_far=0.9429 gen 3: best_reward_so_far=0.9429 ``` Training progress and logs showing sep-CMA-ES generations and the best training reward so far This whole run, embedding included, finished in **6 seconds** on a single auto-selected GPU. That `best_reward_so_far` number is the head's fit to the 35 training rows as CMA-ES optimizes, not a held-out score. What the trained head actually generalizes to is what [Benchmark](#3-benchmark) below measures. ### 5. Register the head A completed job's **Use In Benchmarks** panel offers **Register as Model**. Click it, and the head becomes selectable anywhere BenchGen lets you pick a model, carrying a **Head** badge and a stable name (`head-Qwen3-1.7B-e99b449fde` for this run) so it is never confused with a regular fine-tuned model. There is no merge or export step, since a router head has no adapter weights to fold into a base model. A completed and registered router head, with its head-Qwen3-1.7B-e99b449fde identifier *** ## 3. Benchmark A router head's own output is a routing decision, not an answer. Running it through a benchmark scores the **router and its pool together**, not the head in isolation, since every response a user sees comes from whichever pool member the head picked. BenchGen Router Lite was evaluated against **TR Benchmark (TR, Karma Zorluk)**, an existing BenchGen environment picked deliberately because it is nothing like the training data: 70 Turkish multiple-choice questions, 50 easy and medium general-knowledge questions plus 20 genuinely hard ones (multi-step math, logic, obscure facts), each with four options and a single correct letter. Scoring is deterministic exact-match on the letter, with no LLM-as-judge involved, and because the model under test is a router, the results table also records which pool member answered each question. The TR Benchmark overview page describing the 70-question Turkish benchmark and its scoring rules This is an out-of-distribution test on purpose. The reward dataset was built from English math, knowledge, and reasoning sources; this benchmark is Turkish general knowledge and logic. Evaluating here checks whether the router's routing decision holds up outside the exact domain it learned from, not just inside it. ### Running the evaluation From the benchmark's **Evaluate** tab, the registered head shows up under **Platform Models** alongside every other trained model, carrying its **Head** badge. Selecting it auto-fills the sampling parameters (temperature 0.7, max tokens 150, top P 1.0, 30 second timeout) from the model, and **Run Evaluation** starts the same round trip described above: each question gets embedded, the head picks a pool member, and that pool member answers for real. Selecting the registered benchgen-router-lite head under Platform Models, with sampling parameters auto-filled The evaluation running live, streaming step-by-step progress through the 70 questions ### Watching the questions arrive at the pool Because the router never generates text itself, every question it routes turns into a real API call to the selected pool member. Those calls show up in the OpenRouter dashboard as the run proceeds, which is the clearest way to confirm the routing is genuinely happening rather than being simulated. The generations log lists each call with the model that received it, the provider that served it, token counts, and cost. The three pool members that were reachable during this run appear here by name: `gpt-oss-120b`, `DeepSeek V4 Flash 0731`, and `Ling-3.0-flash`. OpenRouter logs listing generations by model, provider, token counts, and cost Opening a single call shows the exact prompt the pool member received: the benchmark question, its four options, and the instruction to answer with a single letter. This is the benchmark's own formatting, passed through untouched by the router. OpenRouter prompt view showing a multiple choice benchmark question and the answer format instruction The generation details confirm which key issued the call, `benchgen-router-lite`, along with the routing and generation timings and the prompt and completion token counts for that single question. OpenRouter generation details showing the benchgen-router-lite API key, provider routing timings, and token counts ### Results **Overall score: 72.9%** (51 of 70 correct, 1 empty response), broken down by difficulty: | Difficulty | Score | | ---------- | ------------: | | Easy | 78.0% (39/50) | | Medium | 84.6% (11/13) | | Hard | 14.3% (1/7) | The routing distribution tells the more interesting story: **69 of 70 questions were routed to `open_mid`** (`mistralai/mistral-nemo`). The one remaining question came back with no routing decision recorded at all, which is also the run's single empty response. Score breakdown and detailed per-question results, showing the 72.9% overall score and the routing distribution *** ## Explore it on BenchGen The dataset and the trained head both live on the BenchGen platform, where you can inspect their details, schema, and files. Each page also links onward to the underlying repository if you want to download the raw artifacts. 46 tasks, 5 agents, 3 repetitions. The reward matrix the router head trains against, paired with the 1,110-task prompt pool it joins on `task_id`. The trained `head-Qwen3-1.7B-e99b449fde` head: a 10,245-parameter weight vector, its manifest, and a reference implementation of the embed and pick step.