Evaluate an Inference Model

Once a model is running as an inference endpoint, you can benchmark it against any environment in Eval. BenchGen runs the model over every test case in the environment, scores each response, and produces a results report with downloadable predictions and a breakdown for every item.

​

Prerequisites

• A model in the running state. See Deploy an inference model.
• A benchmark or environment to evaluate against, either from the Environments Hub or a custom environment you uploaded.

​

Steps

​

1. Open a benchmark

Open the environment you want to evaluate against. Its Overview tab describes what the benchmark measures and how submissions are scored, and shows tabs for Phases, Leaderboard, Evaluations, and Evaluate. To start, click Evaluate in the top right corner, or open the Evaluate tab.

​

2. Choose a model source

The Evaluate tab opens with “Select a model to evaluate.” Models are grouped by source. Pick the tab that matches where your model lives:

​

3. Select your running model

Since you just deployed an endpoint, click the Running tab. It lists every model that is currently live. Find the one you deployed. It shows a green running badge. Click the model to select it. A checkmark appears and the Run Evaluation button becomes active.

​

4. Run the evaluation

Click Run Evaluation. BenchGen creates an evaluation run, generates a submission for the selected environment, and starts running your model against the test cases.

​

5. Monitor progress

The run opens to a live log view. Status messages stream as the run progresses: it loads the benchmark data, runs the model on each item, and reports progress like Processing: 10/100 (10%). The Evaluation details panel on the right sums up the run: As the run nears completion, the logs show predictions being generated and the final score being computed, for example accuracy=26.00% correct=26/100.

​

6. Review scores and results

When the run finishes, the status turns to Completed and a Score Breakdown replaces the live logs. The headline Overall Score sits at the top, followed by the individual metrics: Below the metrics, Detailed Results shows a table with one row per test case. Each row lists the item ID, the gold (expected) answer, the model’s prediction, and whether it was correct.

​

Download the artifacts

The Files section in the panel on the right lets you download everything the run produced:

​

Next Steps