Skip to main content
​
October 1, 2026
BenchGen agent
  • Benchmarks in your own git repository. Connect a GitHub or GitLab repository in chat with its address and an access token, and the agent saves your benchmark there: questions, scoring and settings, each version a commit by BenchGen Agent. Change a benchmark in the repository, open a pull request for review, list its versions and put an earlier one back live. Private repositories work, and the token is stored encrypted, remembered across chats and never shown back. Benchmarks in git
  • Create a benchmark from a repository. Paste a link to a benchmark in your repository and the agent creates it on the platform. A saved benchmark passes the same checks as an upload, so save, change and create again works end to end. Create a benchmark
  • Open benchmarks from Hugging Face. Ask for “a benchmark from GSM8K” or paste a Hugging Face dataset link. The agent checks the licence before building anything, takes a sample of the test split, keeps the original’s way of scoring and credits the source on the benchmark page. GSM8K, MMLU, ARC, TruthfulQA and similar datasets work. Adapt an open benchmark
  • Fairer scoring of number answers. Models that show their working before the answer are now scored on the number after “Final answer”, or the last number, and “70,000” counts as 70000. The agent’s local rehearsal now tries answers with the working shown first, so a scorer that reads the wrong number is caught before a real run.
  • Compare benchmarks. Ask the agent to compare two benchmarks and it reads both: questions, language, sample size, scoring columns and scores, and tells you how they differ.
  • Find benchmarks by name. Refer to a benchmark by its name, part of it, or its page link. When several share a title, the agent tells them apart by what you can see (published or draft, when it was made, how often it was run) and asks which one you mean.
  • One message, the whole loop. The agent tells you in chat when a run or training job finishes or fails, and when credits run low, so one message can run benchmark, train on the failures, benchmark again and compare. A clear request launches right away, with the price shown first. The autonomous loop
Platform and API
  • Your git repositories. Connect, check and disconnect repositories through the API, and create or update a benchmark from one, public or private, with the new repo:read, repo:write and repo:use scopes. API introduction
  • Readable web addresses. New benchmarks get addresses made from their title. Existing addresses do not change, so links you already shared keep working.
  • Billing. Agent chat is billed per turn and training on the tokens it really uses. You get a notice when your balance runs low.
Web app
  • Diagnostics. A Diagnostics page scans an agent’s conversations every 5 minutes. Ask the agent for a fix and Review with the agent turn what it finds into a proposal, and with the agent’s repository connected in Settings, a proposal becomes a pull request.
  • Agent chat on the Billing page. Agent chat has its own line, next to runs and training.
  • Repository scopes explained. The platform tokens page says what each repo: scope allows.
  • Fixes. Reply blocks no longer go missing in chat, switching agents opens the right chat, and the billing chart shows the right figures.
Docs
​
September 24, 2026
The BenchGen AI agent can now do the benchmark work you do in the web app, by chat.BenchGen agent
  • Create benchmarks in chat. Describe a question-answer or multiple-choice benchmark and the agent builds it, checks it and publishes it when you say so. For anything else (tool calls, custom metrics, several phases) it builds a full bundle and rehearses it locally first, answering everything right and then wrong, so a scorer that marks everything correct is caught before a real run. Create a benchmark
  • Run and follow benchmarks. Pick a model and a benchmark and the agent launches the run, then tells you its state and score. Run a benchmark
  • Edit benchmarks. Rename a benchmark, change its text, pages and questions, replace its scoring or data in place, add a scoring type as a new leaderboard column, and change settings, phases and the ranking column. Every change is checked first and shown before and after. Edit a benchmark
  • Analyse results question by question. See which questions a model got wrong, with the whole answer and what the scoring found missing, and which questions every model misses. Test a scoring fix on a run’s saved answers before changing anything. Analyze results
  • Train from chat. List GPUs, plan and launch training jobs, and follow their logs. Train a model
  • Choices as plain text. The agent asks its questions as numbered lists, never as forms that can stall the chat.
Platform and API
  • Benchmark API. Create a benchmark from a JSON spec or a bundle zip, edit it, publish it, run it and follow the run with run_status and my_runs. Every write takes ?dry_run=1 to check first. API introduction
  • Scopes and limits. Platform tokens get separate benchmark:create, benchmark:edit and benchmark:publish scopes. API tokens and agents can create 10 benchmarks a day and 3 a minute; the web app is not limited.
  • Safer bundle uploads. An upload is refused when its competition.yaml names a file the zip does not hold, and warns about what is valid but probably wrong, such as a leaderboard column no scoring program writes.
  • Per-question results. Read a run question by question, compare it with another run, and see every question across a benchmark’s recent runs.
  • Leaderboard columns, settings and phases. Add, retitle, re-sort and hide columns, and change settings and phases, each through a checked endpoint.
  • Each benchmark has its own web address, and the agent gives you the same link as the API.
Web app
  • Scope hints. The platform tokens page explains each new benchmark scope.
  • Simpler Train menu. Datasets and Models are listed under Train, and Train settings are hidden.
Docs
  • Agent skills section. One page per skill, with what the agent can and cannot do. Skills overview
  • API pages for creating, editing, publishing and running benchmarks, bundle checks, replacing files, leaderboard columns and per-question results.
  • RLVR training. Train with RLVR
​
September 10, 2026
Platform and API
  • Model descriptions. Change the description of a platform model.
  • One key per run. Each benchmark run gets its own access key.
Web app
  • New sidebar. One left menu with Browse and Chat tabs.
  • Shareable links on evaluation and training job pages.
  • Readable model names. Models show their display name everywhere instead of an internal name.
  • Password reset. The reset page shows the real password requirements, and the rate-limit error is clearer.
  • Mobile. A branded mobile header, a sticky header that stays put, and easier tabs.
  • Downloads. Clearer errors and status messages when a merged model is checked or downloaded.
Docs
​
August 31, 2026
BenchGen agent
  • The BenchGen AI agent arrives. A platform agent with its own workspace, available as a default chat, that knows the difference between the BenchGen platform and itself.
Platform and API
  • BenchGen Router Lite. Train a small router head that picks the right agent for each question, try it live, register it as a model and call it through an OpenAI-compatible endpoint, with usage and cost tracked per call. BenchGen Router Lite
  • RL training presets. New orchestrator and conductor presets, with a pool of models from OpenRouter and your own API key per training run.
  • More reliable training. A retried request no longer starts the same training job twice, and a job that lost track of its worker finds it again.
  • Datasets. Create datasets on the platform.
  • Password reset goes through the web app with a styled email, and also updates your chat password.
  • Fair leaderboards. Scores on different scales are normalised before ranking.
Web app
  • Onboarding for new users.
  • Training. Live merge logs on the job page, and a trained LoRA adapter is registered as an export automatically.
  • Router heads. A Try it live panel and one-click registration for finished router head jobs.
  • Payments. A new payment page.
  • Benchmarks. Delete a benchmark permanently, with confirmation. The leaderboard works on mobile and a row opens the model’s page.
  • Models. Suggested models on the not-found page, and a sticky call to action on model and benchmark pages on mobile.
  • Sign-in. The last email you used is remembered.
  • Images in pages open full screen.
Docs
  • BenchGen Router Lite guide with an animated architecture diagram, the Router Fidelity Benchmark results and a cost comparison with frontier models.
  • OpenClaw agents documentation, kept up to date.
​
July 31, 2026
Platform and API
  • Environment-variable benchmarks. A benchmark can declare the environment variables a run needs, with fixed values the owner keeps secret. Runs also take sampling settings and a timeout.
  • Benchmark any model. Run benchmarks against external OpenAI-compatible endpoints and Hugging Face models, not only models trained on the platform, through one run endpoint.
  • RL training. Train with reinforcement learning (RLVR), continue from an earlier RL model, and follow reward and score charts as it trains.
  • Training loops. Set a goal on a benchmark and the platform trains, benchmarks and repeats until the goal is met, optionally deleting the models it no longer needs.
  • LoRA training as its own training type.
  • Billing. Top up credits by card. Training, merging and model calls are billed by the tokens and minutes they really use.
  • Traces. Every agent call is traced, and you can read the traces in the web app.
  • OpenClaw agents. Connect an OpenClaw agent to BenchGen, or disconnect it.
Web app
  • Run settings. An advanced run settings window for environment variables, sampling and timeout, and an Environment tab on each benchmark.
  • External and Hugging Face models. Add an OpenAI-compatible endpoint or search Hugging Face and benchmark the model.
  • Training Loops page with rounds, scores and a goal form.
  • Evaluations. Cancel and delete evaluations, and manage leaderboard columns.
  • Credits. Top-up screen, and credits shown in user management.
  • Model and dataset pages get readable addresses and paging that search engines can follow.
  • Delete an agent.
Docs
​
June 30, 2026
Platform and API
  • Public benchmarks, models and runs. Benchmarks, evaluations, models and training jobs can be public, and visitors can browse them without an account.
  • Benchmark editing for owners. Create a draft benchmark, edit its leaderboard and phases, and add a leaderboard row by hand. Premium Academic users can create benchmarks.
  • Credit check. A run or training job that would go over your credits is stopped before it starts, with a clear message.
  • Logs. Live logs while a run is running, the full log once it finishes, and training logs kept after the job ends.
  • Hugging Face models can be registered on the platform.
  • Training jobs list with paging and search.
Web app
  • Real web addresses for every page, rendered on the server so search engines and link previews see the content, with a sitemap and structured data.
  • Create and edit benchmarks in the web app. A creation flow with Excel import, and an edit page with tabs for details, pages, phases, leaderboard, participation and administrators. Benchmark files can be downloaded.
  • New Datasets and Models pages with a public library, details, owners and visibility toggles.
  • Evaluations. Live logs for running evaluations, the real reason a failed one failed, a link to the dataset generated from a run, and self-reported or verified results.
  • Training jobs. Rename a job, try it again, and download its logs.
  • Run inference. Start a model on a GPU you pick, and use running models in benchmarks.
  • Mobile. Every page works on a phone.
  • Hermes agents can be connected and registered.
Docs
​
May 22, 2026
The platform becomes BenchGen: evaluation and training in one place.Platform and API
  • Benchmark any registered model. Queue a benchmark run for a model registered on the platform, a fine-tuned model or a remote model.
  • Train from a benchmark. A run can produce a training dataset from its answers, ready for fine-tuning.
  • Fine-tuning connected to the platform, with model merging and downloads of merged models.
  • Cancel a run while it is running.
  • GPU pool. Runs share GPUs fairly, prepare in parallel, and recover on their own when a run gets stuck.
  • Public or private models.
Web app
  • Fine-tuning. Train LoRA adapters on Hugging Face datasets, pick a GPU or let BenchGen pick one, follow progress and logs, merge and download the model, continue training an earlier model, and push the result to BenchGen.
  • Explore benchmarks. A benchmark hub with search, and benchmark pages with an overview, leaderboard with search and comparison, and evaluations.
  • Evaluations. A page for every evaluation with its score breakdown and detailed results.
  • OpenClaw actions. An agent action can pass messages to an OpenClaw agent.
Docs
  • The BenchGen docs site launches, with Agents, Eval and Train sections.
​
March 25, 2026
Platform and API
  • Credits. Every run checks your credits first. Users can request credits, and admins grant or subtract them.
  • Leaderboards. Download a public leaderboard as CSV or JSON, choose which columns show, and see analytics charts.
  • Benchmark order. Admins set the order benchmarks appear in.
  • More reliable runs. Runs wait in a queue and go one at a time, and a run stuck after a restart recovers on its own.
  • Large models. Upload progress with a cancel button, and model details read from a large zip without downloading all of it.
  • AI judge is optional for question-answer and multiple-choice benchmarks.
  • File names in any language download correctly.
Web app
  • Agent actions stay in sync with their tools.
​
February 12, 2026
Platform and API
  • Image benchmarks.
  • Single sign-on. Sign in with an organisation’s identity provider, with roles kept in sync at every sign-in.
  • Security hardening. Safer file uploads, protected accounts and profiles, rate limiting, strong passwords, a protected admin panel, and internal addresses never shown to users.
  • Credits for runs, managed by admins.
  • Queues and workers. Manage run queues and workers, check GPU availability before a queue is created, and run several queues in parallel.
  • Leaderboards. Score thresholds, a model column and sortable columns.
  • Question-answer benchmarks from Excel without a context column, or with questions only.
  • Audit log of who did what, with export.
  • API documentation that shows each role the endpoints it can use.
  • Model sharing. Several runs can use the same model at once.
​
December 24, 2025
The benchmark platform.Platform and API
  • Benchmarks for language models. Create benchmarks, run models on them and rank them on a leaderboard.
  • Question-answer and multiple-choice benchmarks, created from an Excel file with a preview first.
  • AI judge. Score answers with a language model you choose.
  • Models. A Models page, uploads of large model files, and an upload API.
  • Invitation codes for joining and for uploading models.
  • Built-in documentation and help throughout, in English and Turkish.
  • Benchmark, not competition. The interface now says benchmark everywhere.
Web app
  • Private templates stay visible to their owner.
  • Agent-to-agent. An agent’s landing page shows its A2A address and instructions.
​
November 26, 2025
Web app
  • Connections. Connect GitHub, Google Workspace, Slack and other services with OAuth, and use a connection to authenticate an agent’s API actions.
  • API testing. Test any API from the browser, through a proxy when the API blocks browser requests, with clear error messages.
  • Single sign-on into chat.
  • Default agent. Messages always reach the right agent and chat.
​
October 17, 2025
Web app
  • Chat on the home page with a default agent, suggested prompts and file uploads.
  • Agent settings. Save a system prompt and suggested prompts, choose the model, and start or stop a custom model.
  • Templates. Deep research, content writer, coder, flight planner, appointment scheduling, customer support, lead qualification and order processing templates, with versions, complexity levels and a duplicate check.
  • Profile. Change your username and password.
  • Stay signed in. Sessions refresh themselves, with a Keep me logged in option.
  • Faster sidebar. Agents load in 2 to 3 seconds instead of over 12.
  • Security. Only an agent’s owner can open, edit or delete it, and deleting an agent removes its chats.
​
August 15, 2025
The first release: build AI agents and chat with them.Web app
  • Agents from templates, with topics, actions and their own instructions.
  • Topics. Create topics yourself or generate them with AI, also from a knowledge base.
  • Actions. Connect an agent to any API, test the call and save it as a template.
  • Knowledge bases. Add documents, Q&A pairs, text and websites (a whole sitemap at once), and edit or delete what was indexed.
  • Agent-to-agent configuration.
  • Chat built on Open WebUI.
Last modified on October 2, 2026