> ## Documentation Index
> Fetch the complete documentation index at: https://benchgen.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Create a benchmark from chat

> The skill that writes a new benchmark from a topic you describe, adapts an open benchmark such as GSM8K or MMLU, or uploads one you already have, and publishes it when you ask.

| The skill | `benchmark-create` |
| - | - |
| **Changes anything** | Yes. It creates a benchmark on your account, and publishes it when you ask. |
| **It picks this up when you say** | "create a benchmark about X", "make a benchmark from GSM8K", "build a benchmark from my questions", "upload a bundle", "publish it" |
| **It hands over to** | `benchmark-launch` to run the new benchmark, `benchmark-edit` to change it later |

## What it can do

* Write a benchmark from a topic, a question count and a format you pick
* Ask questions and expect one short answer, or offer options and expect one letter
* Test whether a model picks the right tool and the right arguments
* Expect a number, right within a tolerance you set
* Score a free answer on the key points it mentions
* Build the whole evaluation package for a format the standard editor cannot express
* Adapt an open benchmark from Hugging Face, such as GSM8K, MMLU or TriviaQA, with its licence checked and its source credited
* Upload a benchmark you already have as a zip
* Build one from a public git repository, so the benchmark has a history
* Drop duplicate questions and spread the correct letters across A, B, C and D
* Rehearse a custom format against a stand-in model, all right, all wrong, and right with the working shown first, before it reaches your account
* Show every question with its answer for approval, then create the benchmark as a draft
* Publish a draft when you ask

![Chat with the BenchGen AI agent: a request for a 10-question multiple choice benchmark, the agent's preview of the questions and answers, the user's confirmation, and the agent's report that the benchmark was created as a draft with a link and three numbered next steps](https://mintcdn.com/benchgen-8fc81371/MiEv2r5AKjiUikhl/images/guides/benchgen-ai-agent/01-create-benchmark.png?fit=max\&auto=format\&n=MiEv2r5AKjiUikhl\&q=85\&s=a0538474484c78472522268a1a7e27a9)

From a request to a draft benchmark: the preview of every question with its answer, the confirmation, and the link

## Adapt an open benchmark

Name a well-known benchmark or paste a Hugging Face dataset link, and the agent turns a sample of
it into a benchmark you can run on any model here. The questions stay the original ones.

**1. It reads the dataset**

The fields, the splits, and the licence the dataset declares. For a benchmark with one part
per subject, such as MMLU, it asks which subject.

**2. It checks the licence**

An open licence such as MIT or Apache 2.0 is fine to publish. For anything else, including a
dataset that declares no licence, it tells you what it found and waits for you to confirm. A
gated dataset cannot be fetched; download it yourself and send the file.

**3. It keeps the original's scoring**

Multiple choice stays multiple choice, with the choices in their original order. A final
number is read after "Final answer", so a model may show its working. A short answer accepts
the original's list of aliases.

**4. It samples the test split**

50 questions by default, up to 2,000. The benchmark page names the source, the split, the rows
used and the licence.

**5. You approve, then it creates**

A preview of sample questions first, as with any benchmark. Afterwards it offers to save the
adapted copy to your repository.

> [!NOTE]
>
> A score on a sample is not the number on the original leaderboard. The agent reports it as "on
> 50 GSM8K test questions", not as a GSM8K score.

## Prompts to try

```text theme={null}
Create a multiple choice benchmark called "Solar System Basics" with 30 questions about
planets, moons, orbits and missions. 4 options each, exactly one correct.
```

```text theme={null}
Create a benchmark that tests whether a model picks the right tool and the right arguments,
with 20 questions about a support agent's tools.
```

```text theme={null}
Make a benchmark from GSM8K with 50 questions from its test split.
```

```text theme={null}
Adapt MMLU high school physics into a benchmark, 30 questions.
```

```text theme={null}
I have a benchmark bundle as a zip. Upload it and create a benchmark from it.
```

```text theme={null}
Create a benchmark from https://github.com/benchgen-ai/benchgen-benchmark-examples/tree/main/benchmarks/cat-care-basics
```

## What it will not do

* Create or publish anything without an explicit request
* Retry on its own. One create per confirmation, and a failure is reported rather than repeated
* Change an existing benchmark, or create a second one to fix a first
* Delete or unpublish anything

> [!NOTE]
>
> Creation is rate limited to 10 benchmarks a day and 3 a minute through the agent. The web app is
> not limited.

## Related

> [card] [Creating a benchmark step by step](https://benchgen.com/docs/guides/benchgen-ai-agent/working-with-the-agent.md#create-a-benchmark)
>
> The walkthrough, with screenshots and large benchmarks.

> [card] [Run a benchmark](https://benchgen.com/docs/skills/run-a-benchmark.md)
>
> Put a model on what you just made.

> [card] [Bundle structure](https://benchgen.com/docs/eval/bundle-structure.md)
>
> What a custom format package contains.

> [card] [Create from a repository (API)](https://benchgen.com/docs/api-reference/endpoint/create-benchmark-repo.md)
>
> Build a benchmark from a bundle in a git repository.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.