Create a Custom Environment

​

What you need

​

Steps

1. Prepare your data. Collect the test cases you want to evaluate against. Package your ground-truth answers into a reference_data.zip.
2. Write a scoring program. Create a script that reads a model’s predictions and compares them to your reference data. The script must write a scores.json file with one numeric value per leaderboard column you want to display.
   {"accuracy": 0.91, "f1": 0.87}
   
   Package the script and a metadata.yaml (specifying the run command) into a scoring_program.zip.
3. (Optional) Write an ingestion program. If BenchGen needs to run your model end-to-end rather than just receive its outputs, write an ingestion program that takes the input data, runs the model, and writes predictions to the output directory. Package it the same way as the scoring program.
4. Write your competition.yaml. This file wires everything together. It declares the title, tasks, which data files to use, and how to display the leaderboard. See the YAML reference for all available fields.
5. Assemble the bundle. Put all files into a single .zip. The competition.yaml must be at the root level.
   my_environment.zip
   ├── competition.yaml
   ├── reference_data.zip
   ├── scoring_program.zip
   ├── ingestion_program.zip   ← optional
   └── input_data.zip          ← optional
   
6. Upload to BenchGen. Go to Environments → New environment → Upload bundle, drop in your .zip, and click Create. BenchGen validates the structure and registers the environment in your workspace immediately.

​

After uploading

• Run it against any model from Eval → Run
• Share it with your team so others can use it in their runs
• Update it by uploading a revised bundle — previous runs keep a snapshot of the version they used

​

Next steps