Skip to main content

​
What it can do

  • Connect the repository you want to use, once, and remember it
  • Record a benchmark’s files in that repository, one commit per change
  • Write the commit message from the change itself, so the log reads as a history
  • Record nothing when the repository already matches, rather than making an empty commit
  • List every version of a benchmark, newest first
  • Put an earlier version back, after a check that says what would change
  • Send only the parts that actually differ, so no run is marked stale for nothing
  • Keep a benchmark’s versions through a retitle, because a version is matched by its number

​
Why it exists

A benchmark on the platform has a timestamp, not a version. The platform can tell you a run is older than the current files. It cannot tell you what changed between them, or give the old files back. Recorded in git, a benchmark gets both. Widening an answer key becomes a diff somebody can read, and a score has a commit to point at instead of a date. The repository is yours. BenchGen does not keep a copy of your benchmarks in a repository of its own, so your questions and answer keys stay in your account, private or public as you choose, and you can revoke our access at any time without asking anyone.

​
Connecting your repository

The first time you ask to save a benchmark, the agent asks for two things: the repository address, and an access token that may write to it.
Make the narrow kind of token. On GitHub that is a fine-grained personal access token limited to that one repository, with Contents: read and write, and an expiry date. On GitLab, a project access token with the api scope. It is no harder to create, and it decides what a leak costs you. The token is checked against the repository before it is stored, so the agent tells you straight away if it cannot write, rather than halfway through saving your first benchmark. After that it is held encrypted, and the agent never shows it again.
A token you paste into a chat is in that conversation’s history from then on. If that matters to you, treat the first one as temporary: connect with it, then create a replacement, connect again, and revoke the first. Reconnecting replaces the credential rather than adding a second one.

​
It is remembered

The repository is stored against your account, not against a conversation. Once connected, it is still there in your next chat, tomorrow, and from any device. You are never asked for the token again just because you started a new conversation.

​
Editing in git instead of on the platform

There are two ways to change a benchmark, and they do not meet until you apply. On the platform is the default. The live benchmark changes now, and the agent records the change afterwards. In the repository is for when you want the change as a reviewable diff before it goes live. Say so, and nothing on the platform moves until you ask:
At that point git has the change and the benchmark does not. When you are ready:
Do not edit the working copy yourself with git commands. It is refreshed before every operation, so anything uncommitted there is discarded. Ask the agent to commit, and it goes through the same path.

​
When saving refuses

If you edited in the repository and then ask to save, the agent stops:
That folder holds work the platform has never seen. Saving would overwrite it. Put it on the platform first, or discard it.
That is the protection working, not a bug. Saving copies the platform over the repository, so without this check an edit you made in git would be silently deleted by the act of “saving” it. You are choosing between two things:

​
When your token expires

Tokens expire, get revoked, and lose access when someone changes a repository’s settings. The agent tells you which repository stopped working and asks for a new token.
Connecting again replaces the old credential in place. It does not make a second connection, and the versions already saved there are untouched. Nothing else has to be set up again. You can also ask before relying on it:

​
Prompts to try

​
Where a benchmark is filed

The name comes from the title, transliterated rather than stripped, so a Turkish or Ukrainian title stays readable instead of losing every letter that is not a to z. The number on the end is what finds a benchmark again after it is renamed, and what keeps two benchmarks with the same title apart. Renaming a benchmark moves the folder rather than filing it twice, and the history follows the move.

​
Who the commits come from

Every version is committed as BenchGen Agent, so a repository full of them reads as one author rather than a stranger, and you can tell at a glance which changes the agent made. That is the author line only. The push is authenticated with your own access token, so the identity has no permissions of its own and no access to your repository beyond the token you gave.

​
What it will not do

  • Create a benchmark, launch a run or start training
  • Put a version back without a check first, which you see before anything changes
  • Show you a stored token again, quote one back to you, or print a repository address that carries one
  • Use a repository you have not named, or invent one when none is connected
  • Keep a copy of your benchmark in a BenchGen repository. There is not one
The history is only as good as the exports. A change that is applied but never recorded is a version that does not exist, which is why the agent records one straight after applying a change, using the same words that described it.

Edit a benchmark

Make the change that is then recorded.

Analyze the results

Decide what to change, with proof, before changing it.

Benchmarks in git

The full guide, with the phrases to use.

Create from a repository (API)

Build a new benchmark from a repository.
Last modified on September 30, 2026