Model Details
BenchGen Router Lite: a lightweight routing model that learns which language model should answer each query, instead of sending every query to the same one.
BenchGen Router Lite (head-Qwen3-1.7B-e99b449fde)
Repo: benchgen/benchgen-router-lite
A router head, not a fine-tuned LLM: a tiny (10,245-parameter) linear
classifier that sits on top of a frozen Qwen/Qwen3-1.7B backbone and learns
which model in a pool should answer a given query. The backbone never
updates — only this head does. Trained with separable CMA-ES (a
gradient-free evolutionary search) against a measured reward matrix, not
gradient descent, and never calls any pool model during training.
Full write-up (dataset, training run, benchmark results): BenchGen Router Lite docs guide.
How it works
query --> frozen Qwen3-1.7B (mean-pooled embedding, 2048-d)
--> this head (linear: logits = embedding @ W + b)
--> argmax over 5 pool agents
--> picked agent's REAL reply is what gets returnedThe router itself never generates an answer — it only picks who should.
Pool (5 agents this head was trained to route between)
| Slot | Agent id (agent_order) | Underlying model | Role |
|---|---|---|---|
| Frontier | frontier_a | openai/gpt-oss-120b | Reasoning |
| Frontier | frontier_b | deepseek/deepseek-v4-flash-0731 | Reasoning |
| Frontier | frontier_c | google/gemma-3-27b-it | Direct |
| Mid-tier | open_mid | mistralai/mistral-nemo | Direct (deliberately the weakest agent) |
| Cheap reasoning | open_cheap_reasoning | inclusionai/ling-3.0-flash | Reasoning |
The underlying model slugs above are not baked into the head — they're
the OpenRouter pool this specific run was trained against. A caller resolves
picked_agent to a real model separately (BenchGen's gateway does this via
its own pool config).
Files
| File | What it is |
|---|---|
head_weights.npy | Flat float32 parameter vector (10,245 values) — theta |
manifest.json | Architecture + training metadata (backbone id, agent order, dims, scores) |
router_head.py | Standalone reference implementation: embed a query, run the head, argmax |
Usage
pip install torch transformers numpy
python router_head.py "What is the derivative of x^3 + 2x?"from router_head import pick_agent
result = pick_agent("What is the derivative of x^3 + 2x?")
print(result["picked_agent"]) # e.g. "frontier_a"manifest.json's head_hidden_dim is 0 for this run, so the head is a
pure linear layer: logits = embedding @ W + b, W shape (2048, 5), b
shape (5,), unflattened from head_weights.npy in that order. The
embedding step (mean-pool the backbone's last hidden state over real,
non-padding tokens) must match exactly — see router_head.py's
embed_query().
Training data & held-out results
Trained on benchgen/router-pilot
(reward matrix: 46 tasks x 5 agents x 3 repetitions, correctness measured, not
assumed) joined with benchgen/router-pilot-tasks
(prompt text) by task_id. Split 35 train / 11 held-out test rows.
| Metric | Score |
|---|---|
| Train reward (CMA-ES fit) | 0.9429 |
| Held-out test reward | 0.8182 |
| Held-out random-agent baseline | 0.6364 |
| Held-out best-single-fixed-agent baseline | 0.7879 |
| Held-out per-question oracle (upper bound) | 0.8485 |
| Beats best fixed agent? | Yes |
The router beats always-calling-the-best-single-agent on held-out data — the actual test of whether the extra routing step is worth it over a naive "always use one model" strategy.
Limitations
- In-distribution only. The reward dataset is English math/knowledge/ reasoning tasks (MATH500, MMLU, MMLU-Pro, ARC-Challenge, GSM8K, AIME2025). Evaluated outside that distribution (a different language, or a very different domain), the classifier's embeddings fall outside anything it learned to discriminate and its pick becomes closer to arbitrary than a real routing decision.
- Pool-specific. This head only knows how to choose between the exact 5 agents listed above, in that exact order. Retraining is required for a different pool.
- Only 46 tasks currently carry reward labels (of 1,110 published prompts in
router-pilot-tasks), so held-out numbers are directional, not a large-n benchmark result.