# Decision Index ![Category](https://img.shields.io/badge/Category-Decision%20Models-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Chance--corrected%20score-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-38%20sub--benchmarks-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) > **Quick answer:** Decision Index (version 0.2.1) is a community-maintained suite that scores "decision models" — models that read a state plus a schema of typed questions and return a calibrated probability for every option in a single forward pass. Results are a chance-corrected aggregate over 38 sub-benchmarks (0 = random guessing, 100 = perfect). Cloudflare's Clef leads at 61.21, self-reported. ## At a Glance **What it tests:** How accurately and how well-calibrated a model is when it must make bounded, structured decisions (choice, score, true/false questions) rather than write free-form text — across tool selection, intent classification, NLI, reasoning, retrieval and safety sub-benchmarks. **Why it matters:** Decision models are a new class of small, fast classifiers designed to sit inside agent workflows. Decision Index gives them a shared yardstick that also reports latency, so accuracy can be weighed against speed. **Known limitations:** The suite is community-run and unofficial. Clef and Clef-Flash were run by their authors (Cloudflare) on 36 of 38 benchmarks, with the missing ones scored as 0, and have not been reproduced by the upstream board. ## What Decision Index Measures Decision models, popularised by TypeSafe AI's Jev, return typed answers — a named choice, an ordinal score, or a true/false probability — for every question asked about a given state. Decision Index aggregates dozens of existing public benchmarks (BFCL, ToolRet, API-Bank, BANKING77, CLINC150, MMLU, GPQA Diamond, GSM8K, MuSR, ForecastBench and others) adapted to this answer format. Scores are chance-corrected so that a model guessing at random scores 0 and a perfect model scores 100, which keeps easy and hard sub-benchmarks comparable. Per-benchmark breakdowns and median/p95 request latency are reported alongside the aggregate. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Decision / classification | | Metric | Chance-corrected aggregate score (0-100) | | Version | 0.2.1 | | Sub-benchmarks | 38 | | Created by | multimodalart and community contributors | | Project | [Hugging Face Space](https://huggingface.co/spaces/multimodalart/jev-decision-index) | ## How Decision Index Is Scored Each sub-benchmark's metric (accuracy, macro-F1, nDCG@10, Brier score) is converted to a chance-corrected score, and the aggregate averages across categories. Latency (median and p95 ms per request) is reported separately and does not enter the score. ## State-of-the-Art Results | Rank | Model | Decision Index | Source | Date | |---|---|---|---|---| | 1 | [Clef](/models/cloudflare/clef) | 61.21 (self-reported) | [Cloudflare decision leaderboard](https://clef-evals.workers-ai-mle.workers.dev/) | 2026-10 | | 2 | Jev (TypeSafe AI, closed API) | 57.91 | [Cloudflare decision leaderboard](https://clef-evals.workers-ai-mle.workers.dev/) | 2026-10 | | 3 | Surogate Rune 26B-A4B v3 | 57.44 | [Cloudflare decision leaderboard](https://clef-evals.workers-ai-mle.workers.dev/) | 2026-10 | | 4 | Decider chat (Gemma-4-31B) | 57.33 | [Cloudflare decision leaderboard](https://clef-evals.workers-ai-mle.workers.dev/) | 2026-10 | | 5 | [Clef-Flash](/models/cloudflare/clef-flash) | 57.07 (self-reported) | [Cloudflare decision leaderboard](https://clef-evals.workers-ai-mle.workers.dev/) | 2026-10 | *Community results snapshot 28 September 2026, mirrored on Cloudflare's leaderboard (built 1 October 2026). Only models with a Benchgen page are linked; the rest are shown for context.* ## Decision Index on Benchgen > No Benchgen results yet — be the first to run Decision Index. ## Decision Index vs Other Benchmarks | Benchmark | What it tests | Output format | |---|---|---| | **Decision Index** | Calibrated structured decisions across 38 sub-benchmarks | Typed probabilities | | [JevBench](/benchmarks/fstandhartinger/jevbench) | Decision accuracy and calibration on a fixed item set | Typed probabilities | | [BFCL v4](/benchmarks/patil-et-al-uc-berkeley/bfcl-v4) | Function-calling accuracy | Generated tool calls | ## Run Decision Index on Your Model Benchgen can track decision-model performance over time alongside the community-reported scores above. ## Related Models & Benchmarks - [Clef](/models/cloudflare/clef) and [Clef-Flash](/models/cloudflare/clef-flash) - [K2-Type-0.9B](/models/ifm/k2-type-0-9b) - [JevBench](/benchmarks/fstandhartinger/jevbench) ## Frequently Asked Questions

What is the Decision Index?

A community suite that scores decision models — models that turn a state and typed questions into calibrated probabilities — as a chance-corrected aggregate over 38 sub-benchmarks.

What does a good Decision Index score look like?

Clef leads at 61.21 (self-reported) and TypeSafe AI's closed Jev scores 57.91; most small open decision models score between roughly 25 and 55.

Who created the Decision Index?

It is maintained by multimodalart and community contributors on Hugging Face, and is unofficial and not affiliated with TypeSafe AI.

Decision Index — Results

Decision Index — Evaluations