PinchBench

PinchBench

Quick answer: PinchBench is a real-world agent benchmark that measures how well LLMs perform as the reasoning "brain" of an OpenClaw coding agent — running actual tasks like scheduling meetings, triaging email, researching topics, writing code, and managing files rather than synthetic academic questions. Scores are the percentage of tasks completed successfully, graded by a mix of automated checks and LLM judges. As of August 2026, Google's Gemini 3 Flash (Preview) leads the verified leaderboard at 95.1%, with Anthropic's Claude Sonnet 4.5 close behind at 93.6%.

At a Glance

What it tests: An LLM's ability to act as the decision-making core of an autonomous OpenClaw agent across eight real-world task categories: productivity (calendar/scheduling), research, writing, coding, analysis, email, memory, and skill discovery.

Why it matters: Most benchmarks isolate a single capability (math, code, knowledge). PinchBench instead scores whether a model can chain tool calls together, handle ambiguous or incomplete instructions, and produce a verifiably correct real-world outcome — closer to how a model behaves once it's deployed as an agent rather than a chat assistant.

Known limitations: Runs go through the OpenClaw harness and OpenRouter routing by default, so scores partly reflect harness/tool-integration behavior in addition to raw model capability. The public leaderboard mixes official (--official-key) and community-submitted runs; Benchgen's leaderboard below uses only verified, official submissions on the current benchmark version to avoid noisy self-reported entries.

What PinchBench Measures

PinchBench (built by the team at Kilo) evaluates language models on 148 tasks spanning calendar and scheduling, web research and data synthesis, content writing, code generation and file operations, spreadsheet/PDF analysis, email triage and search, long-term memory/context retrieval, and OpenClaw skill-ecosystem discovery. A smaller, 53-task open subset ships in the public pinchbench/skill repository so anyone can reproduce a local run; the full 148-task suite backing the live leaderboard is run centrally against submitted models.

Each task is graded automatically, by an LLM judge, or by a hybrid of both — for example, confirming a calendar event was actually created with the correct time, or that an email was correctly filtered and summarized. The benchmark explicitly optimizes for practical, verifiable outcomes ("did it actually create the file, send the email, or schedule the meeting?") rather than proxy metrics like plan quality or reasoning-trace coherence.

Benchmark Specifications

How PinchBench Is Scored

Each submitted run reports a per-task pass/fail (or partial credit for hybrid-graded tasks), aggregated into an overall percentage of tasks completed. The public leaderboard tracks both a model's best score across all its submitted runs and its average score across runs — Benchgen's leaderboard below uses the average score from official, verified submissions on the current (v2.0.0) benchmark version, since a single best-case run can be noisy for an 148-task real-world suite.

State-of-the-Art Results

Scores sourced from the public PinchBench leaderboard (api.pinchbench.com, official + verified submissions, benchmark version 2.0.0) as of August 2026, plus NVIDIA's own reported PinchBench score for Nemotron 3.5 Lightning. Full standings for 20+ additional models are tracked in Benchgen's live leaderboard below.

PinchBench on Benchgen

No Benchgen results yet — be the first to run PinchBench.

PinchBench vs Other Benchmarks

PinchBench differs from most agentic benchmarks by grading everyday personal/professional-assistant workflows (calendar, email, research, file management) rather than pure software-engineering or single-domain tasks — it's a broader test of an agent's practical usefulness across a typical knowledge worker's task mix.

Run PinchBench on Your Model

Because Benchgen tracks version-controlled, regression-tested results over time, running PinchBench through Benchgen lets you catch practical agent-capability regressions between model or harness updates — not just a one-time leaderboard snapshot. This is especially relevant for PinchBench given how much of its score depends on tool-calling reliability and multi-step task execution, both of which are sensitive to prompt, harness, and fine-tuning changes that a single static benchmark run can miss.

Related Models & Benchmarks

• Compare: Gemini 3 Flash · Claude Sonnet 4.5 · Nemotron 3.5 Lightning 30B A3B · Nemotron 3 Super 120B A12B
• Benchmarks: TerminalBench 2.1 · τ³ Banking · SWE-bench Verified

Frequently Asked Questions

Benchmark description based on Benchgen's own analysis of the public PinchBench GitHub repository and API. Last updated 2026-08-11.