> ## Documentation Index
> Fetch the complete documentation index at: https://benchgen.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Create a benchmark from a repository
> Build a benchmark from a bundle that lives in a public git repository, so the benchmark has a history: a commit to point at when someone asks which version of it a score was measured against. Send JSON naming the repository, optionally a branch, tag or commit in `ref`, and optionally the folder inside the repository in `path` when one repository holds several benchmarks. A browse URL works too, so `https://github.com/acme/benchmarks/tree/main/benchmarks/97-horses` needs no `ref` or `path`. The platform reads the owner, repository and ref out of the URL and builds the archive URL itself against an allowed host (github.com and gitlab.com by default), so the fetch cannot be pointed at an address of the caller's choosing. A git host wraps its archive in one top-level folder; that wrapper and the `path` folder are stripped, so `competition.yaml` ends up in the root exactly as an upload needs it. From there the bundle meets the same checks as `create_from_bundle` and is reported through `creation_status`. Private repositories are not supported yet and answer 400. Requires the `benchmark:create` scope and charges the same creation limits; a dry run does not count.
The same bundle format as [create from a bundle](/docs/api-reference/endpoint/create-benchmark-bundle),
read from a public git repository instead of uploaded. This is how a benchmark
gets a history: a change to an answer key becomes a diff somebody can read, and
a score has a commit to point at rather than only a timestamp.
Paste the link you already have. A browse URL carries the branch and the folder
by itself, so `ref` and `path` are only needed for a bare repository address, or
when one repository holds several benchmarks.
There is a working example to try it against, in
[benchgen-ai/benchgen-benchmark-examples](https://github.com/benchgen-ai/benchgen-benchmark-examples):
```bash theme={null}
curl -X POST "https://api.benchgen.com/api/competitions/create_from_repo/?dry_run=1" \
-H "Authorization: Bearer $BENCHGEN_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"repo": "https://github.com/benchgen-ai/benchgen-benchmark-examples/tree/main/benchmarks/cat-care-basics"}'
```
`cat-care-basics` is an open answer benchmark scored on the key points each
answer mentions. Drop the `?dry_run=1` to create it.
## What the platform does with the address
It never fetches the URL you send. It reads the owner, the repository and the
ref out of it and builds the archive address itself, against a host on its
allowlist (`github.com` and `gitlab.com` by default), so the request cannot be
aimed somewhere else. A git host wraps its archive in one top-level folder,
while `competition.yaml` has to sit in the zip root, so that wrapper and the
folder you named are stripped before anything else looks at it.
From there the bundle meets exactly the same checks as an upload, and is
reported through [creation status](/docs/api-reference/endpoint/creation-status).
Private repositories are not supported yet and answer `400` with a plain
explanation rather than a confusing `404`.
Add `?dry_run=1` to validate the bundle and see the title the platform would
use, without creating anything. It does not count against your creation
limits.
## A repository of several benchmarks
One repository can hold many, a folder each. That is how the example repository
is laid out:
```
benchmarks/
cat-care-basics/
competition.yaml
pages/overview.md
terms.md
logo.png
ingestion_program/
scoring_program/
phase/input_data/
phase/reference_data/
```
Point `path` at the folder that holds `competition.yaml`, not at the repository
root. A folder that is not there is answered with the list of folders that are,
so a typo tells you what you meant.
## OpenAPI
````yaml POST /api/competitions/create_from_repo/
openapi: 3.0.3
info:
title: BenchGen Platform API
version: 1.0.0
description: >-
One API for the whole BenchGen platform: model catalogue and serving,
fine-tuning and benchmarks, datasets (knowledge), and billing.
## Authentication
Every request uses the same credential: a platform API token sent as
`Authorization: Bearer bgn_...`.
Create tokens in the web app under Profile Settings > Platform API tokens
(the secret is shown exactly once), or via `POST /api/tokens/` with an
interactive session. Revoking a token disables it platform-wide within 60
seconds.
## Scopes
A token carries scopes chosen at creation; a request outside the token's
scopes gets `403` with an explanatory message.
| scope | grants |
|---|---|
| `models:read` | read model catalogues, job status, logs, GPU info |
| `models:write` | deploy, train, merge, stop models and jobs |
| `benchmark:read` | read benchmark runs and results |
| `benchmark:run` | launch benchmark runs |
| `benchmark:create` | create benchmarks in your account (drafts, Excel,
bundles, specs); counts toward the creation limit |
| `benchmark:manage` | edit and delete benchmarks you own or collaborate on
|
| `benchmark:publish` | publish and unpublish benchmarks you own or
collaborate on |
| `knowledge:read` | read your datasets and fine-tuning data |
| `knowledge:write` | create, edit and delete datasets and fine-tuning data
|
| `billing:read` | read your balance and usage |
| `agents:chat` | chat with your own agents through the API |
| `agents:manage` | manage your agents, knowledge bases and channels |
| `admin` | everything the account can do (staff accounts only) |
## For agents
This document plus `/api/llms.txt` are the machine-readable entry points.
Responses are JSON. Errors use conventional status codes; the body carries
`error` or `message`. Knowledge endpoints return `[{"data": [...], "meta":
{...}}]`.
The complete auto-generated schema of every endpoint (including internal
ones) lives at `/api/public-docs.json` (Swagger 2.0); this document is the
curated, stable, supported surface.
contact:
url: https://benchgen.com
servers:
- url: https://api.benchgen.com
security:
- platformToken: []
tags:
- name: auth
description: Token introspection for services and integrations
- name: tokens
description: Manage your platform API tokens
- name: models
description: Model catalogue and serving
- name: finetune
description: Fine-tuning jobs, inference deployments, GPUs
- name: knowledge
description: Datasets and fine-tuning data (knowledge API)
- name: billing
description: Balance and usage
- name: agents
description: Chat with your agents (OpenAI-compatible facade)
- name: benchmark
description: Public benchmark (competition) listings.
paths:
/api/competitions/create_from_repo/:
post:
tags:
- benchmark
summary: Create a benchmark from a repository
description: >-
Build a benchmark from a bundle that lives in a public git repository,
so the benchmark has a history: a commit to point at when someone asks
which version of it a score was measured against. Send JSON naming the
repository, optionally a branch, tag or commit in `ref`, and optionally
the folder inside the repository in `path` when one repository holds
several benchmarks. A browse URL works too, so
`https://github.com/acme/benchmarks/tree/main/benchmarks/97-horses`
needs no `ref` or `path`. The platform reads the owner, repository and
ref out of the URL and builds the archive URL itself against an allowed
host (github.com and gitlab.com by default), so the fetch cannot be
pointed at an address of the caller's choosing. A git host wraps its
archive in one top-level folder; that wrapper and the `path` folder are
stripped, so `competition.yaml` ends up in the root exactly as an upload
needs it. From there the bundle meets the same checks as
`create_from_bundle` and is reported through `creation_status`. Private
repositories are not supported yet and answer 400. Requires the
`benchmark:create` scope and charges the same creation limits; a dry run
does not count.
parameters:
- name: dry_run
in: query
required: false
schema:
type: boolean
description: 'Validate only: answers 200 with a summary and creates nothing'
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- repo
properties:
repo:
type: string
description: >-
The repository URL, for example
https://github.com/acme/benchmarks. A browse URL carrying a
branch and folder is accepted.
ref:
type: string
description: >-
Branch, tag or commit. Defaults to the repository's default
branch.
path:
type: string
description: >-
The folder inside the repository holding competition.yaml,
when it is not at the root.
responses:
'200':
description: 'dry_run: the bundle is usable; nothing was created'
content:
application/json:
schema:
type: object
properties:
valid:
type: boolean
title:
type: string
description: The title from competition.yaml
files:
type: integer
bytes:
type: integer
unpacked_bytes:
type: integer
warnings:
type: array
items:
type: string
description: >-
Advice that does not block creation: a leaderboard column
no scoring program appears to write, placeholder text left
in the manifest, an empty image, or a manifest with no
pages.
'201':
description: Creation queued
content:
application/json:
schema:
type: object
properties:
valid:
type: boolean
title:
type: string
files:
type: integer
bytes:
type: integer
unpacked_bytes:
type: integer
status_id:
type: integer
description: Poll GET /api/competitions/{status_id}/creation_status/
warnings:
type: array
items:
type: string
description: >-
Advice that does not block creation: a leaderboard column
no scoring program appears to write, placeholder text left
in the manifest, an empty image, or a manifest with no
pages.
'400':
description: The bundle is not usable; `errors` lists every problem found
content:
application/json:
schema:
type: object
properties:
valid:
type: boolean
errors:
type: array
items:
type: string
'403':
description: >-
The token lacks benchmark:create, or the account role may not create
benchmarks
'429':
description: Creation limit reached; a Retry-After header says when to try again
security:
- platformToken: []
- interactiveToken: []
- session: []
components:
securitySchemes:
platformToken:
type: http
scheme: bearer
bearerFormat: bgn_ opaque token
description: >-
Platform API token created under Profile Settings > Platform API tokens.
Scopes are fixed at creation.
interactiveToken:
type: apiKey
in: header
name: Authorization
description: >-
Interactive token from `POST /api/api-token-auth/`, sent as
`Authorization: Token `. Unlike a platform `bgn_` token this is a
full interactive credential (it is minted from your password), so it may
create and revoke platform tokens. Keep it out of CI; put a scoped
`bgn_` token there instead.
session:
type: apiKey
in: cookie
name: sessionid
description: >-
Interactive browser session, the `sessionid` cookie Django sets when you
sign in to the web app (benchgen.com). To script this endpoint, sign in
in a browser and copy the `sessionid` cookie from DevTools > Application
> Cookies. The recommended path is simply the web app itself: Profile
Settings > Platform API tokens. Platform tokens are deliberately refused
here (403), so a leaked token can never mint successors.
````