The BenchGen AI agent in the platform chat
How to use the BenchGen AI agent
1
Open the chat
Click New Chat in the left sidebar. The BenchGen AI agent is the default agent.
2
Ask in your own words, or paste a prompt
Write what you want as you would to a colleague, or copy one of the prompts below with the
button in its top right corner. Words in
[brackets] are yours to replace.3
Confirm what it shows you
Before it creates, spends or changes anything, the agent shows a preview or a plan. Reply
yes
to go ahead, or say what to change.Start here
Three prompts that show you what the agent can do in under a minute. See what you have. Your benchmarks, runs, models, training jobs and credits in one answer.Create a benchmark
Describe what you want to measure. The agent writes the questions, shows them with the answers, and creates the benchmark as a draft only when you say so. How it works. Multiple choice. The most common kind, balanced answers included.Run and compare models
A run needs a published benchmark. The agent says which model it picked and gives you a link. How it works. Publish and run. Both steps in one message.Understand the results
Read only: these prompts never change anything. How it works. Why that score? The wrong answers, and the reason for each.Fix a benchmark
A low score often means the answer key is too narrow, not that the model is weak. The agent proves a fix on answers it already has before offering it. The full story. Find and prove a fix. No model is called, nothing is changed yet.Train and improve a model
Training holds a GPU and spends credits, so the agent always shows a plan and waits foryes.
How it works.
Train on what it got wrong. A fine-tune from a benchmark run’s mistakes.
Keep benchmarks in git
Every version of a benchmark in your own GitHub or GitLab repository, committed as BenchGen Agent. The full guide. Connect a repository. The agent asks for a narrow access token and tells you how to make one.Your account
Credits. What you have left and what things cost.Tips for better results
Say the kind of benchmark and how to score it
Say the kind of benchmark and how to score it
“Multiple choice”, “open answers scored on key points”, “a single number” or “tool calls”
decides how every answer is marked. If you say nothing, the agent picks, and tells you what it
picked.
Ask for a preview or a plan
Ask for a preview or a plan
The agent previews by default, but saying “show me first” or “do not start yet” makes sure a
long request stops at the point where you can still change it.
Refer to things the way you would to a colleague
Refer to things the way you would to a colleague
“My latest benchmark”, “the horse care one”, “my last run” and a page link all work. When two
benchmarks share a name, the agent tells you how they differ (published or draft, when they
were made, how often they were run) and asks which one you mean.
One task per message for long work
One task per message for long work
A chat turn has a time limit. Large benchmarks are written in parts: when the agent stops after
a batch, reply
continue.Write in any language
Write in any language
The agent answers in the language you write in, previews and findings included.
Explore more
Agent overview
Where the agent lives, what it can see, how it is traced.
Working with the agent
Every task step by step, from a first benchmark to a trained model.
All skills
The capabilities the agent is built from.
What the agent may do
What it changes, what it asks first, what it can never do.