From an External API

The External API source lets you benchmark a model that lives behind your own OpenAI-compatible endpoint, such as OpenRouter, Mistral, OpenAI, or a self-hosted server. You give BenchGen the endpoint URL, model name, and an optional key. The credentials are used only for that run and are injected server-side as environment variables.

---

​

Prerequisites

• A benchmark or environment to evaluate against, from the Environments Hub or a custom environment you uploaded.
• An OpenAI-compatible endpoint URL (for example https://openrouter.ai/api/v1/chat/completions).
• The model name as the provider expects it (for example mistralai/mistral-small-24b-instruct-2501).
• An API key or token, if the endpoint requires one.

---

​

Steps

​

1. Open a benchmark

Open the environment you want to evaluate against. The Overview tab describes what the benchmark measures and how it scores submissions. Click Evaluate in the top right, or open the Evaluate tab.

​

2. Switch to the External API source

On the Evaluate tab, the model sources are grouped into tabs: Platform Models, Running, Trained, HuggingFace, and External API. Click External API.

​

3. Enter the endpoint details

Fill in the endpoint form: Click Use this endpoint. A confirmation appears: Selected: mistralai/mistral-small-24b-instruct-2501.

​

4. Set run values (optional)

Expand Advanced: model environment & parameters to review what the benchmark injects. Model connection values are filled from the endpoint you entered; you fill only the fields the benchmark owner exposed (here TOX_SYSTEM_PROMPT, TOX_MAX_QUESTIONS, and the optional TOX_JUDGE_* keys), and you can adjust the sampling parameters. See Environment variables for how these values are declared and injected. Leave a field blank to fall back to the benchmark defaults.

​

5. Run the evaluation

Click Run Evaluation. BenchGen creates the run and opens a live view. While the endpoint is being wired up, the status reads Deploying and the logs wait for output. Once running, the logs confirm the environment variables that were injected into the container, for example MODEL_API_URL, MODEL_NAME, and the benchmark’s prefixed keys such as TOX_API_URL and TOX_MODEL_NAME, followed by per-item progress like [ADIM] 1/30.

​

6. Review the results

When the run finishes, the status turns Completed and the Score Breakdown appears. The example below scored 96.7% (29 of 30 correct). The Detailed Results table lists each item with its correct label, the model’s answer, and the scoring reason. Download the prediction output, scoring output, or the submission archive from the Files panel. See Read results for a full walkthrough.

---

​

Next steps

• Read results: interpret the score breakdown and detailed table.
• Environment variables: how run values reach the model.
• On a platform model: benchmark a model already on BenchGen.