Skip to main content
A model scored 3 out of 10 on a benchmark about horse care. That looks like a weak model. It was not. The benchmark was rejecting correct answers for using different words, and every model would have scored badly on it. This happens to any benchmark scored on key points, on any platform, and usually nobody finds out: a leaderboard gives you a number, not the reason behind it. Five messages to find out which it is. None of them asks the agent to guess.

​
Is it the model or the benchmark?

Getting this wrong is expensive either way: train against a broken answer key and you teach the model the wrong answer, rewrite a benchmark that was right and you have destroyed the measurement. So start with the benchmark, not the score.
A review walks every part of the benchmark and reports what is likely wrong with the evidence attached: duplicate or near-duplicate questions, an expected answer that is not among the question’s own options, correct answers piled onto one letter, groups too small to mean anything, per-question results switched off, runs scored under earlier files whose scores are no longer comparable to each other.
The BenchGen AI agent reviewing a benchmark and reporting four problems with evidence: three of four finished runs were scored under older files, only one run has per-question results, the all key points column cannot tell a close answer from a way off one, and most misses look like the answer key being too narrow rather than the model being wrong

A review of one benchmark: four findings, each with the evidence behind it

The finding that mattered here: every wrong answer had scored exactly the same, two of three key points found, one missed. Seven questions failing identically is a pattern, not seven coincidences. One cause was producing all of them.
A distribution that is too tidy is itself evidence. Genuine model failures vary: some answers are close, some are nowhere near. When every miss lands on the same number, suspect the scoring before the model.

​
Why was this answer marked wrong?

Each one comes back with what was expected, what the model actually wrote, and what the scoring found missing.
The BenchGen AI agent listing the wrong answers of a run question by question, each with the key points that were expected, the model's full answer, and a score of 67 percent key point coverage and 0 percent on all key points

The wrong answers of one run: the key points expected, the full answer, and the score it got

That last part is the whole point, and it is worth being precise about why. An answer can read as correct to a person and still be missing the exact words the key asks for. If the agent judges by reading the answer, it is guessing, and its guess will often be kinder than the scoring was. Reporting what the scoring itself failed to find is a fact rather than an opinion, and only a fact can be acted on.

​
How key point scoring actually works

You need this to read the findings, and it explains most of the misses people hit. A question’s answer key is a list of points, each of which may offer alternatives separated by a pipe:
An answer earns a point if it contains any one of that point’s alternatives. Two rules decide whether it does:
  • Matching is on whole words. “Intestinal” does not contain “intestine”, and “gradually” does not contain “gradual”. This is the single most common cause of a correct answer scoring zero.
  • Order does not matter, and neither does the rest of the sentence. A point is found or it is not.
So in run 1098: None of those is the model being wrong. All three are the key being narrower than the language.

​
Prove the fix before applying it

“Prove it” is doing real work in that sentence. Without it you get a plausible suggestion. With it, the agent copies the benchmark, makes the change in the copy, and scores the run’s already saved answers again. No model is called, so the test costs nothing and finishes in seconds.
The BenchGen AI agent explaining each wrong answer against the answer key and the scoring rule: flee was not a listed alternative for flight, sniff was not listed for smell, and intestinal does not match intestine because the scoring compares whole words

Every miss explained from what the scoring saw, not from reading the answer

Never widen a key with the model’s own sentence. If the model wrote “narrow the age range” and you add that phrase to the key, the benchmark now fits one model’s phrasing, and every other model is judged against it. Add the idea in general words, “age estimate”, “how old”, not the words that happened to appear in this run.
Two more traps worth knowing, because they quietly ruin a benchmark:
  • Do not add a word that wrong answers also use. “Predator” appears in answers that never mention safety, so adding it to a safety key point would pass answers that deserve to fail.
  • A stem is not a fix. Adding “gradu” to catch both “gradual” and “gradually” does nothing, because matching is on whole words. Add both forms, or change how the scoring matches.

​
What the numbers should look like

The BenchGen AI agent proving a fix on a run's saved answers before applying it: before the fix 3 of 10 right, after the widened key 7 of 10 right, four questions now right, three still wrong for stated reasons, and zero newly wrong

The proof, before anything was applied: 3 of 10 to 7 of 10, with none of the right answers broken

Four questions became right, none broke, and three stayed wrong for stated reasons. Read all three numbers, not the first one: The three that stayed wrong are the healthy part. One answer said “speaking softly” where the key asked for a calm or quiet voice; one never mentioned gradual preparation at all. Those are genuine gaps, and a fix that had swept them up too would have been a benchmark that no longer measures anything.
Nothing has changed on the platform at this point. This is the run’s old answers under a changed rule, which predicts what a fresh run would score. It is not a score, and it never reaches a leaderboard.

​
Apply it and measure for real

The same model, on the same questions, scored 6 out of 10. The model had not improved. The benchmark had stopped punishing it for vocabulary.
Runs that finished before the change keep their old scores. They are not re-scored, because a one line change to an answer key must not quietly spend a model run for every entry on the leaderboard. The agent lists which runs are affected and offers to run those models again, so you can put the whole board on one footing when you choose to.

​
Keep the version

A fix nobody can see afterwards is a fix nobody can review.
The files are recorded as a commit, so the change is a diff a colleague can read in ten seconds:
Later, when somebody asks which version of the benchmark a score belongs to, there is an answer:
This needs a repository configured for your deployment. See versioning a benchmark.

​
How to ask, so you get facts back

The five messages work because of how they are phrased. The difference is not politeness, it is whether the reply can be checked.
Treat it as a red flag rather than a success. A benchmark where everything passes separates nothing. Either the change was too broad, or the questions were too easy to begin with. Ask which questions still fail and why; if the answer is none, the benchmark needs harder questions, not a looser key.
No, and nothing does it for you. Earlier runs keep their scores and are marked as scored under previous files, so you can see which numbers are not comparable. Re-run them when you want the leaderboard on one footing, which matters most before you publish or share it.
Almost never. When every model misses the same question, check its answer key first. That is the signature of a key that is wrong or too narrow, and it is the cheapest thing to check.

​
When it really is the model

Not every low score is the benchmark’s fault, and the sorting happens before anything is proposed. It is likely the model when other models get the same question right, when the answer is factually wrong rather than differently worded, or when it ignored an instruction other models followed. Then the work goes to training instead: train on the questions this run missed, and measure again on the same benchmark.

Working with the agent

The broader guide: creating, running and training from chat.

Analyze benchmark results

What the analysis can do, and the phrases that reach it.

Edit a benchmark

Everything the agent can change, and what it will not.

Version a benchmark

Keep the history, and put an earlier version back.
Last modified on September 28, 2026