> ## Documentation Index > Fetch the complete documentation index at: https://benchgen.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Fix a benchmark that scores wrong > A low score often means the answer key is too narrow, not that the model is weak. How to tell which, prove the fix before applying it, and measure again. A model scored 3 out of 10 on a benchmark about horse care. That looks like a weak model. It was not. The benchmark was rejecting correct answers for using different words, and every model would have scored badly on it. This happens to any benchmark scored on key points, on any platform, and usually nobody finds out: a leaderboard gives you a number, not the reason behind it. Five messages to find out which it is. None of them asks the agent to guess. ## Is it the model or the benchmark? Getting this wrong is expensive either way: train against a broken answer key and you teach the model the wrong answer, rewrite a benchmark that was right and you have destroyed the measurement. So start with the benchmark, not the score. ```text theme={null} Review benchmark 97 and tell me everything that looks wrong with it. ``` A review walks every part of the benchmark and reports what is likely wrong **with the evidence attached**: duplicate or near-duplicate questions, an expected answer that is not among the question's own options, correct answers piled onto one letter, groups too small to mean anything, per-question results switched off, runs scored under earlier files whose scores are no longer comparable to each other. ![The BenchGen AI agent reviewing a benchmark and reporting four problems with evidence: three of four finished runs were scored under older files, only one run has per-question results, the all key points column cannot tell a close answer from a way off one, and most misses look like the answer key being too narrow rather than the model being wrong](https://mintcdn.com/benchgen-8fc81371/o4N0GNFEJXrQAuwu/images/guides/benchgen-ai-agent/05-review-findings.png?fit=max\&auto=format\&n=o4N0GNFEJXrQAuwu\&q=85\&s=b8676682a3208c231607540765e73479) A review of one benchmark: four findings, each with the evidence behind it The finding that mattered here: every wrong answer had scored **exactly the same**, two of three key points found, one missed. Seven questions failing identically is a pattern, not seven coincidences. One cause was producing all of them. > [!TIP] > > A distribution that is too tidy is itself evidence. Genuine model failures > vary: some answers are close, some are nowhere near. When every miss lands on > the same number, suspect the scoring before the model. ## Why was this answer marked wrong? ```text theme={null} Show me only the wrong answers of run 1098. ``` Each one comes back with what was expected, what the model actually wrote, and **what the scoring found missing**. ![The BenchGen AI agent listing the wrong answers of a run question by question, each with the key points that were expected, the model's full answer, and a score of 67 percent key point coverage and 0 percent on all key points](https://mintcdn.com/benchgen-8fc81371/o4N0GNFEJXrQAuwu/images/guides/benchgen-ai-agent/06-wrong-answers.png?fit=max\&auto=format\&n=o4N0GNFEJXrQAuwu\&q=85\&s=372401dff1d4b9a46e1086e0895f4f8f) The wrong answers of one run: the key points expected, the full answer, and the score it got That last part is the whole point, and it is worth being precise about why. An answer can read as correct to a person and still be missing the exact words the key asks for. If the agent judges by reading the answer, it is guessing, and its guess will often be kinder than the scoring was. Reporting what the scoring itself failed to find is a fact rather than an opinion, and only a fact can be acted on. ## How key point scoring actually works You need this to read the findings, and it explains most of the misses people hit. A question's answer key is a list of points, each of which may offer alternatives separated by a pipe: ```json theme={null} { "key_points": ["stay|lock apparatus", "flight|escape|danger", "REM|deep sleep"] } ``` An answer earns a point if it contains **any one** of that point's alternatives. Two rules decide whether it does: * **Matching is on whole words.** "Intestinal" does not contain "intestine", and "gradually" does not contain "gradual". This is the single most common cause of a correct answer scoring zero. * **Order does not matter**, and neither does the rest of the sentence. A point is found or it is not. So in run 1098: | The model wrote | The key asked for | Why it scored zero | | - | - | - | | "ready to flee predators" | `flight\|escape\|danger` | "flee" was not one of the alternatives | | "letting the horse sniff it" | `hand\|let it see\|smell` | "sniff" was not listed | | "intestinal twists" | `digest\|intestine\|gut` | whole words, so "intestinal" is not "intestine" | None of those is the model being wrong. All three are the key being narrower than the language. ## Prove the fix before applying it ```text theme={null} Analyse run 1098's wrong answers, and prove any scoring fix you suggest before offering it. ``` "Prove it" is doing real work in that sentence. Without it you get a plausible suggestion. With it, the agent copies the benchmark, makes the change in the copy, and scores the run's **already saved answers** again. No model is called, so the test costs nothing and finishes in seconds. ![The BenchGen AI agent explaining each wrong answer against the answer key and the scoring rule: flee was not a listed alternative for flight, sniff was not listed for smell, and intestinal does not match intestine because the scoring compares whole words](https://mintcdn.com/benchgen-8fc81371/o4N0GNFEJXrQAuwu/images/guides/benchgen-ai-agent/07-why-each-miss.png?fit=max\&auto=format\&n=o4N0GNFEJXrQAuwu\&q=85\&s=c29f56f7424df71774e8ae8513a893f9) Every miss explained from what the scoring saw, not from reading the answer > [!WARNING] > > **Never widen a key with the model's own sentence.** If the model wrote > "narrow the age range" and you add that phrase to the key, the benchmark now > fits one model's phrasing, and every other model is judged against it. Add the > idea in general words, "age estimate", "how old", not the words that happened > to appear in this run. Two more traps worth knowing, because they quietly ruin a benchmark: * **Do not add a word that wrong answers also use.** "Predator" appears in answers that never mention safety, so adding it to a safety key point would pass answers that deserve to fail. * **A stem is not a fix.** Adding "gradu" to catch both "gradual" and "gradually" does nothing, because matching is on whole words. Add both forms, or change how the scoring matches. ## What the numbers should look like ```text theme={null} Both. Prove the whole thing with one check and tell me the before and after. ``` ![The BenchGen AI agent proving a fix on a run's saved answers before applying it: before the fix 3 of 10 right, after the widened key 7 of 10 right, four questions now right, three still wrong for stated reasons, and zero newly wrong](https://mintcdn.com/benchgen-8fc81371/o4N0GNFEJXrQAuwu/images/guides/benchgen-ai-agent/08-proof-before-after.png?fit=max\&auto=format\&n=o4N0GNFEJXrQAuwu\&q=85\&s=e7c7a064b97620f1736c4b2cbb6755a3) The proof, before anything was applied: 3 of 10 to 7 of 10, with none of the right answers broken Four questions became right, **none broke**, and three stayed wrong for stated reasons. Read all three numbers, not the first one: | What you see | What it means | | - | - | | Some now right, none newly wrong | A real fix | | Some now right, some newly wrong | Not a fix. You traded one set of failures for another | | Everything now right | Suspicious. The scoring has probably stopped discriminating | | Nothing changed | Not a fix, whatever it looked like on paper | The three that stayed wrong are the healthy part. One answer said "speaking softly" where the key asked for a calm or quiet voice; one never mentioned gradual preparation at all. Those are genuine gaps, and a fix that had swept them up too would have been a benchmark that no longer measures anything. > [!WARNING] > > Nothing has changed on the platform at this point. This is the run's old > answers under a changed rule, which predicts what a fresh run would score. It > is not a score, and it never reaches a leaderboard. ## Apply it and measure for real ```text theme={null} Apply it, then run the same model again so we can see the new score. ``` The same model, on the same questions, scored **6 out of 10**. The model had not improved. The benchmark had stopped punishing it for vocabulary. > [!NOTE] > > Runs that finished before the change keep their old scores. They are not > re-scored, because a one line change to an answer key must not quietly spend a > model run for every entry on the leaderboard. The agent lists which runs are > affected and offers to run those models again, so you can put the whole board > on one footing when you choose to. ## Keep the version A fix nobody can see afterwards is a fix nobody can review. ```text theme={null} Save benchmark 97 to the repository. ``` The files are recorded as a commit, so the change is a diff a colleague can read in ten seconds: ```diff theme={null} - "key_points": ["flight|escape"] + "key_points": ["flight|escape|flee"] ``` Later, when somebody asks which version of the benchmark a score belongs to, there is an answer: ```text theme={null} What versions do we have of benchmark 97? ``` ```text theme={null} Go back to the answer key we had before the last change. ``` This needs a repository configured for your deployment. See [versioning a benchmark](/docs/skills/version-a-benchmark.md). ## How to ask, so you get facts back The five messages work because of how they are phrased. The difference is not politeness, it is whether the reply can be checked. | Ask for | Instead of | Why | | - | - | - | | "tell me what looks wrong" | "why is the score low" | Invites evidence, not a theory | | "prove it before offering it" | "what should I change" | Turns a suggestion into a measurement | | "tell me how many break too" | "how many does it fix" | One number alone always flatters | | "show me what the scoring found missing" | "is this answer right" | Asks what happened, not for an opinion | > **What if every answer becomes right after the fix?** > > Treat it as a red flag rather than a success. A benchmark where everything > passes separates nothing. Either the change was too broad, or the questions > were too easy to begin with. Ask which questions still fail and why; if the > answer is none, the benchmark needs harder questions, not a looser key. > **Do I have to re-run every model after a fix?** > > No, and nothing does it for you. Earlier runs keep their scores and are marked > as scored under previous files, so you can see which numbers are not > comparable. Re-run them when you want the leaderboard on one footing, which > matters most before you publish or share it. > **One question is wrong for every model. Is that the model's fault?** > > Almost never. When every model misses the same question, check its answer key > first. That is the signature of a key that is wrong or too narrow, and it is > the cheapest thing to check. ## When it really is the model Not every low score is the benchmark's fault, and the sorting happens before anything is proposed. It is likely the **model** when other models get the same question right, when the answer is factually wrong rather than differently worded, or when it ignored an instruction other models followed. Then the work goes to [training](/docs/skills/train-a-model.md) instead: train on the questions this run missed, and measure again on the same benchmark. ## Related > **[Working with the agent](/docs/guides/benchgen-ai-agent/working-with-the-agent.md)** > > The broader guide: creating, running and training from chat. > **[Analyze benchmark results](/docs/skills/analyze-results.md)** > > What the analysis can do, and the phrases that reach it. > **[Edit a benchmark](/docs/skills/edit-a-benchmark.md)** > > Everything the agent can change, and what it will not. > **[Version a benchmark](/docs/skills/version-a-benchmark.md)** > > Keep the history, and put an earlier version back. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.