> ## Documentation Index
> Fetch the complete documentation index at: https://benchgen.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Version a benchmark in git

> The skill that records a benchmark's files in a git repository after every change, lists the versions, and puts an earlier one back.

| The skill | `benchmark-version` |
| - | - |
| **Changes anything** | Records versions without touching the benchmark. Putting one back does change it, after a check. |
| **It picks this up when you say** | "connect my repo", "save it to github", "version my benchmark", "what versions do we have", "go back to the old version" |
| **It hands over to** | `benchmark-launch` to run a model again after a version is restored |

## What it can do

* Connect the repository you want to use, once, and remember it
* Record a benchmark's files in that repository, one commit per change
* Write the commit message from the change itself, so the log reads as a history
* Record nothing when the repository already matches, rather than making an empty commit
* List every version of a benchmark, newest first
* Put an earlier version back, after a check that says what would change
* Send only the parts that actually differ, so no run is marked stale for nothing
* Keep a benchmark's versions through a retitle, because a version is matched by its number

## Why it exists

A benchmark on the platform has a timestamp, not a version. The platform can
tell you a run is older than the current files. It cannot tell you **what**
changed between them, or give the old files back.

Recorded in git, a benchmark gets both. Widening an answer key becomes a diff
somebody can read, and a score has a commit to point at instead of a date.

The repository is **yours**. BenchGen does not keep a copy of your benchmarks
in a repository of its own, so your questions and answer keys stay in your
account, private or public as you choose, and you can revoke our access at any
time without asking anyone.

## Connecting your repository

The first time you ask to save a benchmark, the agent asks for two things: the
repository address, and an access token that may write to it.

```text theme={null}
Save my benchmark to https://github.com/acme/benchmarks
```

Make the narrow kind of token. On GitHub that is a **fine-grained personal
access token** limited to that one repository, with **Contents: read and
write**, and an expiry date. On GitLab, a **project access token** with the
`api` scope. It is no harder to create, and it decides what a leak costs you.

The token is checked against the repository before it is stored, so the agent
tells you straight away if it cannot write, rather than halfway through saving
your first benchmark. After that it is held encrypted, and the agent never
shows it again.

> [!WARNING]
>
> A token you paste into a chat is in that conversation's history from then on.
> If that matters to you, treat the first one as temporary: connect with it,
> then create a replacement, connect again, and revoke the first. Reconnecting
> replaces the credential rather than adding a second one.

## It is remembered

The repository is stored against your account, not against a conversation. Once
connected, it is still there in your next chat, tomorrow, and from any device.
You are never asked for the token again just because you started a new
conversation.

```text theme={null}
Which repository are my benchmarks being saved to?
```

## Editing in git instead of on the platform

There are two ways to change a benchmark, and they do not meet until you apply.

**On the platform** is the default. The live benchmark changes now, and the
agent records the change afterwards.

**In the repository** is for when you want the change as a reviewable diff
before it goes live. Say so, and nothing on the platform moves until you ask:

```text theme={null}
I want to edit this in my repo, not on the platform.
```

```text theme={null}
Widen the answer key on question 3 so "flee" counts, and commit it to my repo.
```

At that point git has the change and the benchmark does not. When you are
ready:

```text theme={null}
Show me what applying my repo would change on the benchmark, then do it.
```

> [!WARNING]
>
> Do not edit the working copy yourself with git commands. It is refreshed
> before every operation, so anything uncommitted there is discarded. Ask the
> agent to commit, and it goes through the same path.

## When saving refuses

If you edited in the repository and then ask to save, the agent stops:

> That folder holds work the platform has never seen. Saving would overwrite
> it. Put it on the platform first, or discard it.

That is the protection working, not a bug. Saving copies the platform over the
repository, so without this check an edit you made in git would be silently
deleted by the act of "saving" it. You are choosing between two things:

| You mean | Say |
| - | - |
| The repo version is the newer one | "apply my repo to the benchmark" |
| The repo edit was a mistake | "discard it and take the platform version" |

## When your token expires

Tokens expire, get revoked, and lose access when someone changes a repository's
settings. The agent tells you which repository stopped working and asks for a
new token.

```text theme={null}
My token expired, here is a new one: <token>
```

Connecting again **replaces** the old credential in place. It does not make a
second connection, and the versions already saved there are untouched. Nothing
else has to be set up again.

You can also ask before relying on it:

```text theme={null}
Is my repository still connected?
```

## Prompts to try

```text theme={null}
Save my horse care benchmark to the repository.
```

```text theme={null}
What versions do we have of the horse care benchmark?
```

```text theme={null}
Go back to the answer key we had before the last change.
```

```text theme={null}
What changed in this benchmark since last week?
```

## Where a benchmark is filed

```
benchmarks/<name>-<id>/
    competition.yaml, the scoring program, the questions and the answers
```

The name comes from the title, transliterated rather than stripped, so a
Turkish or Ukrainian title stays readable instead of losing every letter that
is not a to z. The number on the end is what finds a benchmark again after it
is renamed, and what keeps two benchmarks with the same title apart.

Renaming a benchmark **moves** the folder rather than filing it twice, and the
history follows the move.

## Who the commits come from

Every version is committed as **BenchGen Agent**, so a repository full of them
reads as one author rather than a stranger, and you can tell at a glance which
changes the agent made.

That is the author line only. The push is authenticated with your own access
token, so the identity has no permissions of its own and no access to your
repository beyond the token you gave.

## What it will not do

* Create a benchmark, launch a run or start training
* Put a version back without a check first, which you see before anything changes
* Show you a stored token again, quote one back to you, or print a repository
  address that carries one
* Use a repository you have not named, or invent one when none is connected
* Keep a copy of your benchmark in a BenchGen repository. There is not one

> [!NOTE]
>
> The history is only as good as the exports. A change that is applied but never
> recorded is a version that does not exist, which is why the agent records one
> straight after applying a change, using the same words that described it.

## Related

> [card] [Edit a benchmark](https://benchgen.com/docs/skills/edit-a-benchmark.md)
>
> Make the change that is then recorded.

> [card] [Analyze the results](https://benchgen.com/docs/skills/analyze-results.md)
>
> Decide what to change, with proof, before changing it.

> [card] [Benchmarks in git](https://benchgen.com/docs/guides/benchgen-ai-agent/benchmarks-in-git.md)
>
> The full guide, with the phrases to use.

> [card] [Create from a repository (API)](https://benchgen.com/docs/api-reference/endpoint/create-benchmark-repo.md)
>
> Build a new benchmark from a repository.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.