Model Files — Instructions

Model Files — Instructions

Read this before creating, renaming, or syncing any model file.

Slug Rules (CRITICAL)

The filename (without .md) is the canonical slug used in three places:

• Platform URL: benchgen.com/models/{org-slug}/{model-slug}
• Leaderboard entry model: field
• Leaderboard filename: leaderboard/{benchmark}--{model-slug}.md

All three must match exactly. A mismatch creates broken links on every leaderboard entry for that model.

Slug derivation from api_name

The platform derives the URL slug from api_name via:

api_name → lowercase → remove () → replace non-alphanumeric with hyphens → trim hyphens

The filename must equal this derived slug. Verify with:

# Quick check for a single model
echo "GPT-4o (May 2024)" | tr '[:upper:]' '[:lower:]' | sed 's/[()]//g' | sed 's/[^a-z0-9]/-/g' | sed 's/-\{2,\}/-/g' | sed 's/^-\|-$//g'

Example: api_name: Llama 3.1 70B Instruct → llama-3-1-70b-instruct → file must be llama-3-1-70b-instruct.md

Naming Rules

1. Instruct variants — always add -instruct

If the hf_id or official name ends in -Instruct or -Instructed, the filename slug must include -instruct. Check the HuggingFace model card or official release post — not just the short marketing name.

2. MoE models — include active parameter count

If the official name includes an active-params suffix like (550B A55B) or A22B, include it in the slug.

Exception: Always verify the live platform URL before adding any suffix. Some models drop architecture suffixes in their actual platform slug. Open https://benchgen.com/models/{org}/{slug} in a browser to confirm.

3. Org slug — use canonical short names only

Before Creating a New Model File

Run this checklist every time:

1. Derive the slug from the official name using the derivation rule above
2. Check for conflicts: ls models/{slug}.md — must not exist
3. Check hf_id: if it ends in -Instruct or -Instructed, the slug must include -instruct
4. Check MoE suffix: if the official name has active-params like (A22B), include it in the slug
5. Verify org slug: use the table above
6. Set api_name so the derivation rule round-trips to exactly the intended slug
7. Verify platform URL: open https://benchgen.com/models/{org-slug}/{model-slug} — if the page exists, your slug must match it exactly

Sync Notes

• Sync models before leaderboard entries. The model must have entity_id written by the sync script before leaderboard entries can include model_entity_id.
• The BENCHGEN_USER_TOKEN JWT expires in ~24 hours. If you get 401 Unautorizate, fetch a fresh token from DevTools → Network → any knowledge-api request → Request Headers → authorization value.
• Never set entity_id by hand — the sync script writes it automatically.

Pre-flight Audit

Run before any batch create or rename operation:

node --input-type=module << 'EOF'
import { readFileSync, readdirSync } from "fs";
for (const f of readdirSync("models").filter(f => f.endsWith(".md") && f !== "_TEMPLATE.md")) {
  const slug = f.replace(".md", "");
  const src = readFileSync(`models/${f}`, "utf8");
  const apiName = (src.match(/^api_name:\s*(.+)$/m)?.[1] || "").trim().replace(/^["']|["']$/g, "");
  if (!apiName) continue;
  const derived = apiName.toLowerCase().replace(/[()]/g, "").replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, "");
  if (derived !== slug) console.log(`MISMATCH: file=${slug}  api_name="${apiName}"  derived=${derived}`);
}
EOF

See benchmarks/configs/BENCHMARK_WORKFLOW.md for the full end-to-end workflow.