# FullStackBench (Chinese) ![Category](https://img.shields.io/badge/Category-Coding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-pass%401-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-%E2%89%881%2C684-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-medium-d29922) ![Created](https://img.shields.io/badge/Created-Nov%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2412.00535) [![GitHub](https://img.shields.io/badge/GitHub-bytedance%2FFullStackBench-181717?logo=github&logoColor=white)](https://github.com/bytedance/FullStackBench) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/ByteDance/FullStackBench) > **Quick answer:** FullStackBench (Chinese) is the Chinese-instruction configuration of ByteDance Seed's FullStackBench dataset — the same 16-language, multi-domain full-stack coding tasks as the English split, but with native Chinese-language task instructions, testing whether a model's coding ability holds up when the prompt itself is not in English. ## At a Glance **What it tests:** Full-stack coding ability across 16 programming languages and real-world domains, specifically when task instructions are given in Chinese rather than English — probing whether coding capability is robust to instruction language or quietly degrades outside English prompts. **Why it matters:** Most code-generation benchmarks (and most model pretraining corpora) skew heavily English; the Chinese-instruction split directly tests a real gap that matters for any deployment serving non-English-speaking developers. **Known limitations:** Like the English split, this requires standing up the SandboxFusion execution sandbox to run; comparing scores directly against the English split can also be confounded by instruction phrasing differences, not purely language. ## What FullStackBench (Chinese) Measures FullStackBench (Chinese) draws on the same underlying task domains as the English split — basic programming, data analysis, software engineering, mathematics, and machine learning — across the same 16 programming languages, but with task instructions authored natively in Chinese rather than translated mechanically from the English set. This design choice mirrors the paper's broader goal of reflecting "real-world usage scenarios rather than simple translations," extended here to the instruction language itself. Grading uses the same **SandboxFusion** execution sandbox as the English split: submissions are run against real unit tests per task, and a submission passes or fails based on actual code execution rather than any language-dependent string matching — meaning the Chinese split measures a model's ability to correctly parse Chinese instructions into working, multilingual code, not just its Chinese-language fluency in isolation. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Coding / full-stack, multilingual, execution-verified (Chinese instructions) | | Metric | pass@1 (execution-based correctness via SandboxFusion) | | Number of tasks | ≈1,684 (Chinese-instruction split of ~3,374 total dataset rows) | | Languages | 16 programming languages (task code); Chinese task instructions | | Saturation | Medium — varies by language, domain, and model's Chinese-instruction comprehension | | Created by | ByteDance Seed Foundation Code Team (Siyao Liu, He Zhu, Jerry Liu, et al.) | | Source paper | [Liu et al. 2024](https://arxiv.org/abs/2412.00535) | | GitHub | [bytedance/FullStackBench](https://github.com/bytedance/FullStackBench) | | Dataset | [ByteDance/FullStackBench](https://huggingface.co/datasets/ByteDance/FullStackBench) | ## How FullStackBench (Chinese) Is Scored Identical grading mechanism to the English split: each task is scored pass@1 via real code execution against unit tests inside the SandboxFusion sandbox. Results are typically reported per programming language and per domain, and are often directly compared against the same model's English-split score to quantify any instruction-language performance gap. ## State-of-the-Art Results > Scores vary substantially by language, domain, and a model's Chinese-instruction comprehension. See the [official GitHub repository](https://github.com/bytedance/FullStackBench) and paper for the full per-model breakdown across both instruction-language splits. ## FullStackBench (Chinese) on Benchgen > No Benchgen results yet — [be the first to run FullStackBench (Chinese)](/benchmarks/bytedance-seed-foundation-code-team/fullstackbench-chinese). ## FullStackBench (Chinese) vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **FullStackBench (Chinese)** | Multi-domain, multilingual full-stack coding (Chinese instructions) | ≈1,684 | medium | | [FullStackBench (English)](/benchmarks/bytedance-seed-foundation-code-team/fullstackbench-english) | Same tasks, English instructions | ≈1,690 | medium | | [RepoBench](/benchmarks/tianyang-liu-canwen-xu-julian-mcauley-uc-san-diego/repobench) | Repository-level code completion | — | medium | FullStackBench (Chinese) shares its underlying task design and grading with the English split, differing only in instruction language — making the pair uniquely useful for directly measuring instruction-language robustness on the same coding tasks. ## Run FullStackBench (Chinese) on Your Model Benchgen lets teams evaluate their own coding model on Chinese-instructed full-stack tasks and compare directly against its English-split score, surfacing any instruction-language gap in coding capability. ## Frequently Asked Questions

What is FullStackBench (Chinese)?

FullStackBench (Chinese) is the Chinese-instruction configuration of ByteDance Seed's FullStackBench dataset, testing full-stack coding ability across 16 languages when task instructions are given in Chinese.

What does a good FullStackBench (Chinese) score look like?

A strong result means matching or closely tracking the same model's English-split pass@1 score, indicating coding capability that doesn't meaningfully degrade when instructions are given in Chinese.

Who created FullStackBench (Chinese)?

FullStackBench, including its Chinese-instruction split, was created by the ByteDance Seed Foundation Code Team, published in late 2024.

FullStackBench (Chinese) — Phases

FullStackBench (Chinese)