Skip to main content
POST
Create a benchmark from a repository
The same bundle format as create from a bundle, read from a public git repository instead of uploaded. This is how a benchmark gets a history: a change to an answer key becomes a diff somebody can read, and a score has a commit to point at rather than only a timestamp. Paste the link you already have. A browse URL carries the branch and the folder by itself, so ref and path are only needed for a bare repository address, or when one repository holds several benchmarks. There is a working example to try it against, in benchgen-ai/benchgen-benchmark-examples:
cat-care-basics is an open answer benchmark scored on the key points each answer mentions. Drop the ?dry_run=1 to create it.

What the platform does with the address

It never fetches the URL you send. It reads the owner, the repository and the ref out of it and builds the archive address itself, against a host on its allowlist (github.com and gitlab.com by default), so the request cannot be aimed somewhere else. A git host wraps its archive in one top-level folder, while competition.yaml has to sit in the zip root, so that wrapper and the folder you named are stripped before anything else looks at it. From there the bundle meets exactly the same checks as an upload, and is reported through creation status.
Private repositories are not supported yet and answer 400 with a plain explanation rather than a confusing 404.
Add ?dry_run=1 to validate the bundle and see the title the platform would use, without creating anything. It does not count against your creation limits.

A repository of several benchmarks

One repository can hold many, a folder each. That is how the example repository is laid out:
Point path at the folder that holds competition.yaml, not at the repository root. A folder that is not there is answered with the list of folders that are, so a typo tells you what you meant.

Authorizations

Authorization
string
header
required

Platform API token created under Profile Settings > Platform API tokens. Scopes are fixed at creation.

Query Parameters

dry_run
boolean

Validate only: answers 200 with a summary and creates nothing

Body

application/json
repo
string
required

The repository URL, for example https://github.com/acme/benchmarks. A browse URL carrying a branch and folder is accepted.

ref
string

Branch, tag or commit. Defaults to the repository's default branch.

path
string

The folder inside the repository holding competition.yaml, when it is not at the root.

Response

dry_run: the bundle is usable; nothing was created

valid
boolean
title
string

The title from competition.yaml

files
integer
bytes
integer
unpacked_bytes
integer
warnings
string[]

Advice that does not block creation: a leaderboard column no scoring program appears to write, placeholder text left in the manifest, an empty image, or a manifest with no pages.

Last modified on September 29, 2026