Skip to main content
The External API source lets you benchmark a model that lives behind your own OpenAI-compatible endpoint, such as OpenRouter, Mistral, OpenAI, or a self-hosted server. You give BenchGen the endpoint URL, model name, and an optional key. The credentials are used only for that run and are injected server-side as environment variables.
When to use this. Reach for External API when the model is served somewhere else behind an OpenAI-compatible API. If the model is on BenchGen use On a platform model; to pull one from the Hub use From a HuggingFace model.

​
Prerequisites

  • A benchmark or environment to evaluate against, from the Environments Hub or a custom environment you uploaded.
  • An OpenAI-compatible endpoint URL (for example https://openrouter.ai/api/v1/chat/completions).
  • The model name as the provider expects it (for example mistralai/mistral-small-24b-instruct-2501).
  • An API key or token, if the endpoint requires one.

​
Steps

​
1. Open a benchmark

Open the environment you want to evaluate against. The Overview tab describes what the benchmark measures and how it scores submissions.
The benchmark overview page

The benchmark overview page

Click Evaluate in the top right, or open the Evaluate tab.

​
2. Switch to the External API source

On the Evaluate tab, the model sources are grouped into tabs: Platform Models, Running, Trained, HuggingFace, and External API. Click External API.
The model source tabs on the Evaluate screen

The model source tabs on the Evaluate screen

​
3. Enter the endpoint details

Fill in the endpoint form:
The External API form filled in for a Mistral model on OpenRouter

The External API form filled in for a Mistral model on OpenRouter

Click Use this endpoint. A confirmation appears: Selected: mistralai/mistral-small-24b-instruct-2501.
Credentials are used only to run this evaluation and are injected into the benchmark container as environment variables. They are not stored as a reusable model.

​
4. Set run values (optional)

Expand Advanced: model environment & parameters to review what the benchmark injects. Model connection values are filled from the endpoint you entered; you fill only the fields the benchmark owner exposed (here TOX_SYSTEM_PROMPT, TOX_MAX_QUESTIONS, and the optional TOX_JUDGE_* keys), and you can adjust the sampling parameters.
The Advanced panel with owner-exposed fields and sampling parameters

The Advanced panel with owner-exposed fields and sampling parameters

See Environment variables for how these values are declared and injected. Leave a field blank to fall back to the benchmark defaults.

​
5. Run the evaluation

Click Run Evaluation. BenchGen creates the run and opens a live view. While the endpoint is being wired up, the status reads Deploying and the logs wait for output.
The run page showing Deploying status

The run page showing Deploying status

Once running, the logs confirm the environment variables that were injected into the container, for example MODEL_API_URL, MODEL_NAME, and the benchmark’s prefixed keys such as TOX_API_URL and TOX_MODEL_NAME, followed by per-item progress like [ADIM] 1/30.
Live logs listing the injected environment variables and run progress

Live logs listing the injected environment variables and run progress

​
6. Review the results

When the run finishes, the status turns Completed and the Score Breakdown appears. The example below scored 96.7% (29 of 30 correct).
The completed run with a 96.7% score and detailed per-item results

The completed run with a 96.7% score and detailed per-item results

The Detailed Results table lists each item with its correct label, the model’s answer, and the scoring reason. Download the prediction output, scoring output, or the submission archive from the Files panel. See Benchmark Results for a full walkthrough.

​
Next steps

Benchmark Results

Interpret the score breakdown and detailed table.

Environment Variables

How run values reach the model.

On a Platform Model

Benchmark a model already on BenchGen.
Last modified on September 3, 2026