Benchgen
Supported agents

Connect the agent you already run.

BenchGen evaluates an agent's behavior, not a specific model or framework. Connect it through a plugin, register it, or point BenchGen at its endpoint, then benchmark it, read its traces, and turn real runs into training data.

Benchmarks, datasets and training in one platformDocumented setup for each pathCloud, or your own Kubernetes with Helm

OpenClaw Agent

Available

Connect an OpenClaw gateway with one plugin

Install the BenchGen plugin in your OpenClaw gateway to get traces of every turn, benchmarks that run against the agent like any model, datasets for training, and chat. The connection is outbound only, so nothing needs to be opened inbound.

  • Plugin: @benchgen/benchgen-openclaw
  • Trace of every turn the agent handles
  • Benchmarks run against the agent like any model
  • Datasets from traces, for training
  • Chat with the agent from BenchGen
  • Outbound-only connection
$ openclaw plugins install @benchgen/benchgen-openclaw
Setup guide

Hermes Agent

Available

Evaluate & fine-tune your Hermes trajectories

Hermes agents can be connected to BenchGen and registered (available since release 0.7.0). The Hermes page covers the evaluation workflow: scoring runs on the five quality dimensions and deciding between a harness fix and a LoRA fine-tune.

  • Connect and register your Hermes agent
  • Five quality dimensions
  • Harness fix vs. fine-tune guidance
  • LoRA fine-tune export
Release notes

Any OpenAI-compatible endpoint

Available

Benchmark any model or agent behind an API

Point BenchGen at the URL of any OpenAI-compatible endpoint, such as OpenRouter, Mistral, OpenAI or a self-hosted server, add the model name and an optional key, and run it against an environment. Credentials are used only for that run.

  • Endpoint URL and model name, nothing else required
  • Runs against any benchmark environment
  • Credentials used for the run only
  • Per-item results and downloadable artifacts
External API guide

Your agent not listed?

Building on a different agent framework? Let us know.

How it works

Three ways to connect an agent.

Which one fits depends on how your agent runs. Once connected, the agent is benchmarked like any other model on BenchGen.

Plugin

OpenClaw runs your agent inside a gateway; the BenchGen plugin connects it. Traces, benchmarks, datasets and chat, with nothing inbound to your gateway.

Connect and register

Hermes agents can be connected and registered with BenchGen, so they can be selected and evaluated alongside other models.

Endpoint

Anything served behind an OpenAI-compatible endpoint can be benchmarked through the External API source, without installing anything.

Prefer to drive it from code? The BenchGen API takes one platform token for benchmarks, runs, datasets and training. Read the API reference.