Skip to main content
The BenchGen AI agent is the default agent in the platform chat. It works as you: it creates benchmarks, launches runs, and starts training jobs on your account, with your permissions and your credits. New to it? See the overview for where to open the chat, or the prompt library for prompts to paste.
The agent cannot stop or delete a training job, and it cannot delete or unpublish a benchmark. Do those from the web app.

Create a benchmark

Describe what the benchmark should measure. The agent writes the questions and shows you a preview with the correct answers. Nothing exists yet: reply Create it to create the benchmark, or say what to change.
It supports question and answer benchmarks, multiple choice benchmarks, and custom formats (tool calling, number answers, open answers scored on key points) that it builds as a bundle. The benchmark is created as a draft. The agent reports its id and link, then asks what to do next: publish it, run a model on it, or leave it as a draft.
A draft cannot be run. If you ask for a run on a draft, the agent tells you so and offers to publish it first. To do both in one step, say “publish it and run <model name> on it”.
Chat with the BenchGen AI agent: a request for a 10-question multiple choice benchmark, the agent's preview of the questions and answers, the user's confirmation, and the agent's report that the benchmark was created as a draft with a link and three numbered next steps

Creating a benchmark: the request, the preview, the confirmation, and the draft with its link and next steps

You do not need to ask for the correct answers to be spread across A, B, C and D. The agent’s tooling reorders the options so the correct letters come out even.

Large benchmarks

A benchmark can hold up to 2,000 questions. One chat turn is limited to about 15 minutes, so the agent writes a large benchmark in parts of about 25 questions, checks them for duplicates, and submits them as a single benchmark.
Around 150 questions fit in one turn. For more, the agent stops after a batch, tells you how many questions are written, and continues when you reply continue.
A benchmark keeps the evaluation and scoring programs it was created with. Platform improvements to those programs apply to benchmarks created afterwards, so compare scores within one benchmark rather than across an old one and a new one.

Run a benchmark on a model

The benchmark must be published. Use a model name from the catalogue or one of your own models; if several models match, the agent says which one it picked. It replies with the run id and a link. When you ask for the status later, it reports the overall score and the breakdown by category. To compare models, run the same benchmark on each and ask:

Look inside a run

A finished run of a question and answer or multiple choice benchmark produces a dataset. Ask the agent to show it:
Chat with the BenchGen AI agent showing one dataset row in mode distill: the full multiple choice prompt with its answer format instruction, followed by the benchmarked model's reasoning and its final answer D

One row in mode distill: the exact prompt the benchmark sent, and the strong model's own answer with its reasoning. This pair is what a distilled model learns from.

Mode all is the fastest way to understand a surprising score. If a model scores lower than you expect, read what it actually answered before drawing conclusions: a low score is often an answer format problem, such as a missing final answer line, rather than missing knowledge.

Analyze a run and fix the benchmark

A low score is not always the model’s fault. Before you train anything, ask what actually went wrong: an answer key that rejects correct answers in other words will make every model look bad, and training against it teaches the wrong answer.
The agent explains each miss from what the scoring saw rather than from reading the answer, and it proves a fix on the run’s already saved answers before offering it, so nothing is guessed and no model is called. In one case a model went from 3 out of 10 to 6 out of 10 with no question made easier: the benchmark had been rejecting correct answers for using different words.

Fix a benchmark by talking to it

The whole conversation, message by message, with the proof and the before and after.

Train a model

The agent can launch LoRA fine-tuning, reinforcement learning (GRPO), continued training of an existing adapter, and training on the results of a benchmark run. Training holds a GPU and spends credits, so it is always two steps:
1

Ask for a plan

The agent shows exactly what it would submit: job type, base model, dataset, hyperparameters, and GPU availability. Nothing has started yet.
2

Confirm in your next message

Reply yes. Anything else cancels. To change something, say what to change and ask for the updated plan first.
3

Follow the job

Ask training status or training logs. When the job completes, the trained model is registered in your catalogue and you can run benchmarks on it.
The base model for training is a Hugging Face repository id such as Qwen/Qwen2.5-0.5B-Instruct, not a catalogue display name. Check the base_model line in the plan before you confirm.

Close the full improvement loop

The most useful workflow chains the steps above: measure a model, train it on what the benchmark revealed, and measure again.
1

Benchmark the model

Run a benchmark on the model you want to improve and wait for the score.
2

Train on the run

By default this trains on the questions the model got wrong. Confirm with yes.
3

Benchmark the trained model

4

Compare

Ask for both scores side by side. The agent does not claim an improvement before the second run has finished.

Distill a strong model into a small one

You can also teach a small open-source model from a strong model’s run. The benchmark records how the strong model reasoned on every question. Distillation keeps the answers it got right and trains the small model on that reasoning.
1

Run the strong model

Run the benchmark on a frontier model. This is the ceiling.
2

Run the small model

Run the same benchmark on the open-source model. This is the baseline.
3

Plan the distillation

The plan states how many correct trajectories the run holds. Always name the base model: the strong model’s run does not say which model to train.
The BenchGen AI agent's training plan: from-run distillation, base model Qwen2.5-0.5B-Instruct, source run 1081, dataset mode distill, 30 correct trajectories, 10 epochs, LoRA rank 16, and a note that nothing has been launched

A distillation plan: source run, dataset mode, the number of correct trajectories, and the hyperparameters. Nothing is launched until you confirm.

4

Confirm, then benchmark the result

Reply yes, wait for the job to complete, run the same benchmark on the trained model, and ask for the three scores side by side.
In one of our runs on a 30-question multiple choice benchmark, a frontier model scored 100%, a 0.5B open-source model scored 46.7%, and the same 0.5B model scored 73.3% after distillation.
A table from the BenchGen AI agent comparing three runs on the same benchmark: the frontier model at 100, the base 0.5B model at 46.7, and the distilled 0.5B model at 73.3, each with its run link, followed by a summary of the 26.7 point gain

The result of one distillation loop, reported by the agent: the strong model, the base model, and the distilled model on the same benchmark

Three things make distillation work well:
  • Match the language. A small model that scores at guessing level on a benchmark, for example 25% with four options, has little to build on. Use a benchmark language the small model reads well.
  • Give it capacity. A larger student, such as a 1.5B model, gains more and more steadily.
  • Check on new questions. Training and testing on the same questions includes memorisation. For an honest number, create a second benchmark on the same topic with different questions and compare the base and distilled models there.

Merge and publish a trained model

A trained adapter is served on top of its base model by default, and that is enough to benchmark it. Merging is an explicit export, for when you want standalone weights:
The agent states the model id and the name it will get, asks for confirmation, and publishes the merged model to your catalogue once the merge is available.

Limits and troubleshooting

Tips for talking to the agent
  • Be specific: give names, counts, and ids (“30 questions”, “run 1081”).
  • Ask for one action per message. Each step that spends credits gets its own confirmation.
  • When it asks a numbered question, reply with the number.
  • Runs and training take minutes. Ask “what is the status?” instead of waiting.
  • Start a new chat for a new piece of work.
  • You never paste an API token into the chat, and the agent never prints one.

Agent skills

The capabilities the agent is built from, with what each one covers.

BenchGen AI agent overview

Where to open the chat and what to ask first.

Create a benchmark (API)

The same creation flow, called from your own code.

Run a benchmark (API)

Launch runs and read their status programmatically.

Launch training (API)

Start training jobs without the chat.
Last modified on September 30, 2026