Version a benchmark in a repository you own, review changes as a diff before they go live, and put an earlier answer key back. All from chat.
A benchmark on a platform has a timestamp, not a version. It can tell you a run
is older than the current files. It cannot tell you what changed between
them, or give the old files back.That matters the first time someone asks which version of a benchmark a score
belongs to, and nobody can answer. A leaderboard is a column of numbers whose
meaning quietly changed underneath them.Kept in a repository, a benchmark gets a history you can read. Widening an
answer key stops being an invisible edit and becomes this:
The repository is yours. BenchGen keeps no copy of your benchmarks in a
repository of its own, so your questions and answer keys stay in your account,
private or public as you choose, and you can revoke access whenever you like.
Create an empty repository on GitHub. No files, no README, nothing. Then:
Save my benchmark to https://github.com/me/mybenchmarks
The agent asks for an access token. Make the narrow kind, it is no harder:
GitHub
Settings, Developer settings, fine-grained personal access token. Limit
it to that one repository, give it Contents: read and write, and set an
expiry date. A classic token with the whole repo scope also works and
reaches every repository you own, which is the reason not to use one.
GitLab
A project access token on that project, with the api scope and at
least the Developer role.
The token is checked against the repository before it is stored, so you find
out immediately if it cannot write, rather than halfway through your first
save. After that it is held encrypted and never shown again.
A token pasted into a chat stays in that conversation’s history. Treat the
first one as temporary: connect with it, then create a replacement, send that,
and revoke the first. Reconnecting replaces the credential in place and
nothing you have saved is affected.
You only do this once. The repository is remembered against your account, so it
is still there in your next conversation, tomorrow, and from any device.
This is the part worth understanding, because picking the wrong one wastes the
work.
On the platform
The live benchmark changes now. The agent saves a version afterwards. Use
it when you just want the change made.
In the repository
Git changes, the benchmark does not, until you apply. Use it when you want
to see the change as a diff before anyone runs against it.
For the second, say so plainly:
I want to edit this in my repo, not on the platform.Widen the answer key on question 3 so "flee" counts, and commit it.
Now the change exists as a diff you can open on GitHub and show someone. When
you are happy:
Show me what applying my repo would change on the benchmark, then do it.
Do not edit the agent’s working copy yourself with git commands. It is
refreshed before every operation, so anything uncommitted there is thrown
away. Ask the agent to commit and it goes through the same checked path.
Committing straight to main is fine when the repository is your own notes. As
soon as anyone else cares how a benchmark is scored, a change should be
something they approve, not something they find later.
Widen the answer key so "flee" counts, and open a pull request for it.
You get a branch, a pushed commit and a review link. Open it, read the diff,
merge it when you are happy.
Merging changes the repository, not the benchmark. Applying is still a
separate step, deliberately: a merged pull request should not silently change
what every model on your leaderboard was measured against.
I merged it, put it live.
Opening a review needs one permission beyond pushing: Pull requests: write
on GitHub. If your token does not have it, the branch is still pushed and you
get a link that opens the request with one click. Nothing is lost.
Go back to the answer key we had before the last change.
The agent tells you what would change and waits. Nothing is applied until you
say so. Afterwards, run the model again: old runs keep their old scores and are
marked as scored under previous files, because a one line change to an answer
key must not quietly spend a model run for every entry on your leaderboard.
If you edited in the repository and then ask to save, you will see something
like:
That folder holds work the platform has never seen. Saving would overwrite
it. Put it on the platform first, or discard it.
That is the protection working. Saving copies the platform over the
repository, so without this check the edit you made in git would be deleted by
the act of saving it. You are choosing between two things:
You mean
Say
The repo version is the newer one
”apply my repo to the benchmark”
The repo edit was a mistake
”discard it and take the platform version”
The agent will not choose for you, because one of them destroys work.
The repository is not only somewhere versions go. It is also somewhere
benchmarks come from, which is how you move one between environments, hand one
to a colleague, or rebuild one you archived.
Create a benchmark from https://github.com/me/mybenchmarks/tree/main/benchmarks/horse-care-basics-97
Point at the folder, not the repository, when it holds more than one
benchmark. A GitHub browse link carries the branch and the folder already, so
that one address is everything the agent needs.
This works on a private repository. Once you have connected it, your
stored credential is used for the fetch automatically. You do not need to
make anything public, and you are not asked for the token again.
Because a repository is a normal way in, the round trip closes:
Save my horse care benchmark to my repo. -> it is now a folder with a historyCreate a benchmark from that folder. -> a second copy, from the same files
You can connect several, for example one private repository for real
benchmarks and one public for examples.
Which repositories do I have connected?
Save this one to my public examples repo instead.
Only the first repository you connect becomes your default. When you connect
another one for a job, name it in the same message (“connect this, then save
the MMLU benchmark there”) and the agent uses it for that work. With several
connected and none named, it asks which before saving.A benchmark lives where you first saved it. If you are not sure which
repository holds one, ask.
The agent cannot create a repository for you. Create an empty one at
github.com/new and send its address with a token. That limit is deliberate: a
token that can create repositories can reach every repository you own, which
is exactly what the narrow token avoids.
Every version is committed as BenchGen Agent, so a repository full of them
reads as one author and you can see at a glance which changes the agent made.That is the author line only. The push is authenticated with your own access
token, so the identity has no permissions of its own.
benchmarks/<name>-<id>/ competition.yaml, the scoring program, the questions and the answers
The name comes from the title, transliterated rather than stripped, so a
Turkish or Ukrainian title stays readable instead of losing every letter that
is not a to z. The number on the end is what finds a benchmark again after it
is renamed, and what keeps two benchmarks with the same title apart.Renaming a benchmark moves the folder rather than filing it twice, and the
history follows the move.
