What it can do
- Write a benchmark from a topic, a question count and a format you pick
- Ask questions and expect one short answer, or offer options and expect one letter
- Test whether a model picks the right tool and the right arguments
- Expect a number, right within a tolerance you set
- Score a free answer on the key points it mentions
- Build the whole evaluation package for a format the standard editor cannot express
- Adapt an open benchmark from Hugging Face, such as GSM8K, MMLU or TriviaQA, with its licence checked and its source credited
- Upload a benchmark you already have as a zip
- Build one from a public git repository, so the benchmark has a history
- Drop duplicate questions and spread the correct letters across A, B, C and D
- Rehearse a custom format against a stand-in model, all right, all wrong, and right with the working shown first, before it reaches your account
- Show every question with its answer for approval, then create the benchmark as a draft
- Publish a draft when you ask
From a request to a draft benchmark: the preview of every question with its answer, the confirmation, and the link
Adapt an open benchmark
Name a well-known benchmark or paste a Hugging Face dataset link, and the agent turns a sample of it into a benchmark you can run on any model here. The questions stay the original ones.1
It reads the dataset
The fields, the splits, and the licence the dataset declares. For a benchmark with one part
per subject, such as MMLU, it asks which subject.
2
It checks the licence
An open licence such as MIT or Apache 2.0 is fine to publish. For anything else, including a
dataset that declares no licence, it tells you what it found and waits for you to confirm. A
gated dataset cannot be fetched; download it yourself and send the file.
3
It keeps the original's scoring
Multiple choice stays multiple choice, with the choices in their original order. A final
number is read after “Final answer”, so a model may show its working. A short answer accepts
the original’s list of aliases.
4
It samples the test split
50 questions by default, up to 2,000. The benchmark page names the source, the split, the rows
used and the licence.
5
You approve, then it creates
A preview of sample questions first, as with any benchmark. Afterwards it offers to save the
adapted copy to your repository.
A score on a sample is not the number on the original leaderboard. The agent reports it as “on
50 GSM8K test questions”, not as a GSM8K score.
Prompts to try
What it will not do
- Create or publish anything without an explicit request
- Retry on its own. One create per confirmation, and a failure is reported rather than repeated
- Change an existing benchmark, or create a second one to fix a first
- Delete or unpublish anything
Creation is rate limited to 10 benchmarks a day and 3 a minute through the agent. The web app is
not limited.
Related
Creating a benchmark step by step
The walkthrough, with screenshots and large benchmarks.
Run a benchmark
Put a model on what you just made.
Bundle structure
What a custom format package contains.
Create from a repository (API)
Build a benchmark from a bundle in a git repository.