> ## Documentation Index > Fetch the complete documentation index at: https://benchgen.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Keep your benchmarks in your own GitHub > Version a benchmark in a repository you own, review changes as a diff before they go live, and put an earlier answer key back. All from chat. A benchmark on a platform has a timestamp, not a version. It can tell you a run is older than the current files. It cannot tell you **what** changed between them, or give the old files back. That matters the first time someone asks which version of a benchmark a score belongs to, and nobody can answer. A leaderboard is a column of numbers whose meaning quietly changed underneath them. Kept in a repository, a benchmark gets a history you can read. Widening an answer key stops being an invisible edit and becomes this: ```diff theme={null} - "key_points": ["flight|escape"] + "key_points": ["flight|escape|flee"] ``` The repository is **yours**. BenchGen keeps no copy of your benchmarks in a repository of its own, so your questions and answer keys stay in your account, private or public as you choose, and you can revoke access whenever you like. ## Connect it once Create an empty repository on GitHub. No files, no README, nothing. Then: ```text theme={null} Save my benchmark to https://github.com/me/mybenchmarks ``` The agent asks for an access token. Make the narrow kind, it is no harder: > **GitHub** > > Settings, Developer settings, **fine-grained personal access token**. Limit > it to that one repository, give it **Contents: read and write**, and set an > expiry date. A classic token with the whole `repo` scope also works and > reaches every repository you own, which is the reason not to use one. > **GitLab** > > A **project access token** on that project, with the `api` scope and at > least the Developer role. The token is checked against the repository before it is stored, so you find out immediately if it cannot write, rather than halfway through your first save. After that it is held encrypted and never shown again. > [!WARNING] > > A token pasted into a chat stays in that conversation's history. Treat the > first one as temporary: connect with it, then create a replacement, send that, > and revoke the first. Reconnecting replaces the credential in place and > nothing you have saved is affected. You only do this once. The repository is remembered against your account, so it is still there in your next conversation, tomorrow, and from any device. ## Saving a version Call it whatever you call it. You never need its number. ```text theme={null} Save my horse care benchmark to my repo. ``` You get a commit id and a link. Say what changed and the history becomes worth reading: ```text theme={null} Save it, and say that I widened the answer key so "flee" counts as "flight". ``` > [!TIP] > > Ask for a version straight after any change you might want to undo. A change > that was applied but never saved is a version that does not exist. ## The two ways to change a benchmark This is the part worth understanding, because picking the wrong one wastes the work. > **On the platform** > > The live benchmark changes now. The agent saves a version afterwards. Use > it when you just want the change made. > **In the repository** > > Git changes, the benchmark does not, until you apply. Use it when you want > to see the change as a diff before anyone runs against it. For the second, say so plainly: ```text theme={null} I want to edit this in my repo, not on the platform. Widen the answer key on question 3 so "flee" counts, and commit it. ``` Now the change exists as a diff you can open on GitHub and show someone. When you are happy: ```text theme={null} Show me what applying my repo would change on the benchmark, then do it. ``` > [!WARNING] > > Do not edit the agent's working copy yourself with git commands. It is > refreshed before every operation, so anything uncommitted there is thrown > away. Ask the agent to commit and it goes through the same checked path. ## Putting a change up for review Committing straight to `main` is fine when the repository is your own notes. As soon as anyone else cares how a benchmark is scored, a change should be something they approve, not something they find later. ```text theme={null} Widen the answer key so "flee" counts, and open a pull request for it. ``` You get a branch, a pushed commit and a review link. Open it, read the diff, merge it when you are happy. > [!WARNING] > > Merging changes the **repository**, not the benchmark. Applying is still a > separate step, deliberately: a merged pull request should not silently change > what every model on your leaderboard was measured against. ```text theme={null} I merged it, put it live. ``` > [!NOTE] > > Opening a review needs one permission beyond pushing: **Pull requests: write** > on GitHub. If your token does not have it, the branch is still pushed and you > get a link that opens the request with one click. Nothing is lost. ## Reading the history ```text theme={null} What versions do we have of the horse care benchmark? ``` ```text theme={null} What changed in this benchmark since last week? ``` Each version has a short id, which is what you quote to say which one you mean. ## Going back ```text theme={null} Go back to the answer key we had before the last change. ``` The agent tells you what would change and waits. Nothing is applied until you say so. Afterwards, run the model again: old runs keep their old scores and are marked as scored under previous files, because a one line change to an answer key must not quietly spend a model run for every entry on your leaderboard. ## When saving refuses If you edited in the repository and then ask to save, you will see something like: > That folder holds work the platform has never seen. Saving would overwrite > it. Put it on the platform first, or discard it. **That is the protection working.** Saving copies the platform over the repository, so without this check the edit you made in git would be deleted by the act of saving it. You are choosing between two things: | You mean | Say | | - | - | | The repo version is the newer one | "apply my repo to the benchmark" | | The repo edit was a mistake | "discard it and take the platform version" | The agent will not choose for you, because one of them destroys work. ## Creating a benchmark from your repository The repository is not only somewhere versions go. It is also somewhere benchmarks come from, which is how you move one between environments, hand one to a colleague, or rebuild one you archived. ```text theme={null} Create a benchmark from https://github.com/me/mybenchmarks/tree/main/benchmarks/horse-care-basics-97 ``` Point at the **folder**, not the repository, when it holds more than one benchmark. A GitHub browse link carries the branch and the folder already, so that one address is everything the agent needs. > [!NOTE] > > This works on a **private** repository. Once you have connected it, your > stored credential is used for the fetch automatically. You do not need to > make anything public, and you are not asked for the token again. Because a repository is a normal way in, the round trip closes: ```text theme={null} Save my horse care benchmark to my repo. -> it is now a folder with a history Create a benchmark from that folder. -> a second copy, from the same files ``` ## More than one repository You can connect several, for example one private repository for real benchmarks and one public for examples. ```text theme={null} Which repositories do I have connected? ``` ```text theme={null} Save this one to my public examples repo instead. ``` Only the first repository you connect becomes your default. When you connect another one for a job, name it in the same message ("connect this, then save the MMLU benchmark there") and the agent uses it for that work. With several connected and none named, it asks which before saving. A benchmark lives where you first saved it. If you are not sure which repository holds one, ask. > [!NOTE] > > The agent cannot create a repository for you. Create an empty one at > github.com/new and send its address with a token. That limit is deliberate: a > token that can create repositories can reach every repository you own, which > is exactly what the narrow token avoids. ## Who the commits come from Every version is committed as **BenchGen Agent**, so a repository full of them reads as one author and you can see at a glance which changes the agent made. That is the author line only. The push is authenticated with your own access token, so the identity has no permissions of its own. ## How a benchmark is filed ``` benchmarks/-/ competition.yaml, the scoring program, the questions and the answers ``` The name comes from the title, transliterated rather than stripped, so a Turkish or Ukrainian title stays readable instead of losing every letter that is not a to z. The number on the end is what finds a benchmark again after it is renamed, and what keeps two benchmarks with the same title apart. Renaming a benchmark **moves** the folder rather than filing it twice, and the history follows the move. ## Related > **[Version a benchmark](/docs/skills/version-a-benchmark.md)** > > What the skill can and will not do. > **[Fixing a benchmark](/docs/guides/benchgen-ai-agent/fixing-a-benchmark.md)** > > Find out why a score is low, and prove the fix before applying it. > **[Connect a repository (API)](/docs/api-reference/endpoint/connect-git-repository.md)** > > The same connection, from your own code. > **[Working with the agent](/docs/guides/benchgen-ai-agent/working-with-the-agent.md)** > > Creating, running and training from chat. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.