After you train and save a model, you run inference on it by deploying it from its model card. Running a model loads it onto a GPU and exposes an OpenAI-compatible endpoint you can send requests to. This is the same Run Inference flow used for any model on BenchGen, so the full walkthrough lives in Deploy an inference model.
Prerequisites
Steps
1. Open the model and click Run Inference
Open the model’s card and click Run Inference in the top right.
In the Run Inference dialog, pick a GPU and review the inference settings (max tokens, temperature, top P, context length, precision), then click Run model.
3. Wait for the endpoint to be ready
The status reads Deploying while the model starts up, then turns to running once it’s live. The model card’s endpoint panel fills in with the LiteLLM Name, Endpoint URL, and access token.
For the full deployment walkthrough, including the deployment logs and how to verify the model is serving, see Deploy an inference model.
Send requests
Once the status is running, the endpoint is OpenAI-compatible. Use the LiteLLM Name, Endpoint URL, and Token from the model card:
Next Steps
Last modified on June 26, 2026
