FullStackBench (English)
1 phaseActive
ByteDance Seed's full-stack coding benchmark (English instruction split), covering 16 languages and domains like data analysis, ML, and web dev via SandboxFusion.
Overview
FullStackBench (English)
Quick answer: FullStackBench is a ByteDance Seed benchmark of roughly 3,000 full-stack programming tasks across 16 languages and real-world domains (data analysis, machine learning, software engineering, and more), executed through ByteDance's own SandboxFusion sandbox; this page covers the English-instruction split of the dataset.
At a Glance
What it tests: Broad, execution-verified coding ability across mainstream real-world application domains and 16 programming languages, not just isolated single-function code generation.
Why it matters: Most code benchmarks (HumanEval, MBPP) test narrow, single-function Python problems; FullStackBench instead spans the domains real developers actually work in — data analysis, ML, and general software engineering — across many languages, with tasks designed as genuine real-world instructions rather than simple translations of existing problems.
Known limitations: Because tasks require an actual execution sandbox (SandboxFusion) rather than static string matching, running the full benchmark requires standing up the sandbox server locally or via Docker, a heavier setup than static benchmarks.
What FullStackBench (English) Measures
FullStackBench covers a wide range of application domains — basic programming, data analysis, software engineering, mathematics, and machine learning — deliberately designed to reflect the domains professional developers actually work in, rather than narrow algorithmic puzzles. Tasks are written across 16 widely-used programming languages, with real-world instructions and matching unit tests authored per-language rather than mechanically translated from a single source language, so multilingual results reflect genuine per-language task design rather than translation artifacts.
Correctness is verified using SandboxFusion, ByteDance's open-sourced code execution sandbox that supports many languages and packages behind a standardized HTTP API, incorporating over 10 coding-related evaluation datasets under one uniform interface. The dataset ships with distinct instruction-language configurations — this page covers the English-instruction (en) split, evaluating a model's full-stack coding ability when prompted in English.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Coding / full-stack, multilingual, execution-verified |
| Metric | pass@1 (execution-based correctness via SandboxFusion) |
| Number of tasks | ≈1,690 (English-instruction split of ~3,374 total dataset rows) |
| Languages | 16 programming languages |
| Saturation | Medium — varies substantially by language and application domain |
| Created by | ByteDance Seed Foundation Code Team (Siyao Liu, He Zhu, Jerry Liu, et al.) |
| Source paper | Liu et al. 2024 |
| GitHub | bytedance/FullStackBench |
| Dataset | ByteDance/FullStackBench |
How FullStackBench (English) Is Scored
Each task is graded pass@1 via actual code execution against unit tests inside the SandboxFusion sandbox — a submission either passes all associated tests or it doesn't. Because the dataset spans 16 languages and 5+ application domains, results are typically broken down by language and domain rather than reported as a single aggregate score, since performance can vary substantially across both axes.
State-of-the-Art Results
Scores vary substantially by language and application domain. See the official GitHub repository and paper for the full per-model, per-language, per-domain breakdown.
FullStackBench (English) on Benchgen
No Benchgen results yet — be the first to run FullStackBench (English).
FullStackBench (English) vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| FullStackBench (English) | Multi-domain, multilingual full-stack coding | ≈1,690 | medium |
| HumanEval+ | Rigorous single-function Python generation | 164 | high |
| RepoBench | Repository-level code completion | — | medium |
FullStackBench is distinguished from narrow single-function benchmarks like HumanEval+ by spanning real-world application domains (data analysis, ML, web/desktop dev) across 16 languages rather than isolated algorithmic Python problems.
Run FullStackBench (English) on Your Model
Benchgen lets teams evaluate their own coding model against FullStackBench's English-instruction tasks across 16 languages and real-world domains, surfacing language- and domain-specific weaknesses a single-language benchmark would miss.