FullStackBench (Chinese)

FullStackBench (Chinese)

Quick answer: FullStackBench (Chinese) is the Chinese-instruction configuration of ByteDance Seed's FullStackBench dataset — the same 16-language, multi-domain full-stack coding tasks as the English split, but with native Chinese-language task instructions, testing whether a model's coding ability holds up when the prompt itself is not in English.

At a Glance

What it tests: Full-stack coding ability across 16 programming languages and real-world domains, specifically when task instructions are given in Chinese rather than English — probing whether coding capability is robust to instruction language or quietly degrades outside English prompts.

Why it matters: Most code-generation benchmarks (and most model pretraining corpora) skew heavily English; the Chinese-instruction split directly tests a real gap that matters for any deployment serving non-English-speaking developers.

Known limitations: Like the English split, this requires standing up the SandboxFusion execution sandbox to run; comparing scores directly against the English split can also be confounded by instruction phrasing differences, not purely language.

What FullStackBench (Chinese) Measures

FullStackBench (Chinese) draws on the same underlying task domains as the English split — basic programming, data analysis, software engineering, mathematics, and machine learning — across the same 16 programming languages, but with task instructions authored natively in Chinese rather than translated mechanically from the English set. This design choice mirrors the paper's broader goal of reflecting "real-world usage scenarios rather than simple translations," extended here to the instruction language itself.

Grading uses the same SandboxFusion execution sandbox as the English split: submissions are run against real unit tests per task, and a submission passes or fails based on actual code execution rather than any language-dependent string matching — meaning the Chinese split measures a model's ability to correctly parse Chinese instructions into working, multilingual code, not just its Chinese-language fluency in isolation.

Benchmark Specifications

How FullStackBench (Chinese) Is Scored

Identical grading mechanism to the English split: each task is scored pass@1 via real code execution against unit tests inside the SandboxFusion sandbox. Results are typically reported per programming language and per domain, and are often directly compared against the same model's English-split score to quantify any instruction-language performance gap.

State-of-the-Art Results

Scores vary substantially by language, domain, and a model's Chinese-instruction comprehension. See the official GitHub repository and paper for the full per-model breakdown across both instruction-language splits.

FullStackBench (Chinese) on Benchgen

No Benchgen results yet — be the first to run FullStackBench (Chinese).

FullStackBench (Chinese) vs Other Benchmarks

FullStackBench (Chinese) shares its underlying task design and grading with the English split, differing only in instruction language — making the pair uniquely useful for directly measuring instruction-language robustness on the same coding tasks.

Run FullStackBench (Chinese) on Your Model

Benchgen lets teams evaluate their own coding model on Chinese-instructed full-stack tasks and compare directly against its English-split score, surfacing any instruction-language gap in coding capability.

Frequently Asked Questions