# FullStackBench (English) ![Category](https://img.shields.io/badge/Category-Coding-1f6feb) ![Metric](https://img.shields.io/badge/Metric-pass%401-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-%E2%89%881%2C690-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-medium-d29922) ![Created](https://img.shields.io/badge/Created-Nov%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2412.00535) [![GitHub](https://img.shields.io/badge/GitHub-bytedance%2FFullStackBench-181717?logo=github&logoColor=white)](https://github.com/bytedance/FullStackBench) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/ByteDance/FullStackBench) > **Quick answer:** FullStackBench is a ByteDance Seed benchmark of roughly 3,000 full-stack programming tasks across 16 languages and real-world domains (data analysis, machine learning, software engineering, and more), executed through ByteDance's own SandboxFusion sandbox; this page covers the **English-instruction** split of the dataset. ## At a Glance **What it tests:** Broad, execution-verified coding ability across mainstream real-world application domains and 16 programming languages, not just isolated single-function code generation. **Why it matters:** Most code benchmarks (HumanEval, MBPP) test narrow, single-function Python problems; FullStackBench instead spans the domains real developers actually work in — data analysis, ML, and general software engineering — across many languages, with tasks designed as genuine real-world instructions rather than simple translations of existing problems. **Known limitations:** Because tasks require an actual execution sandbox (SandboxFusion) rather than static string matching, running the full benchmark requires standing up the sandbox server locally or via Docker, a heavier setup than static benchmarks. ## What FullStackBench (English) Measures FullStackBench covers a wide range of application domains — basic programming, data analysis, software engineering, mathematics, and machine learning — deliberately designed to reflect the domains professional developers actually work in, rather than narrow algorithmic puzzles. Tasks are written across 16 widely-used programming languages, with real-world instructions and matching unit tests authored per-language rather than mechanically translated from a single source language, so multilingual results reflect genuine per-language task design rather than translation artifacts. Correctness is verified using **SandboxFusion**, ByteDance's open-sourced code execution sandbox that supports many languages and packages behind a standardized HTTP API, incorporating over 10 coding-related evaluation datasets under one uniform interface. The dataset ships with distinct instruction-language configurations — this page covers the English-instruction (`en`) split, evaluating a model's full-stack coding ability when prompted in English. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Coding / full-stack, multilingual, execution-verified | | Metric | pass@1 (execution-based correctness via SandboxFusion) | | Number of tasks | ≈1,690 (English-instruction split of ~3,374 total dataset rows) | | Languages | 16 programming languages | | Saturation | Medium — varies substantially by language and application domain | | Created by | ByteDance Seed Foundation Code Team (Siyao Liu, He Zhu, Jerry Liu, et al.) | | Source paper | [Liu et al. 2024](https://arxiv.org/abs/2412.00535) | | GitHub | [bytedance/FullStackBench](https://github.com/bytedance/FullStackBench) | | Dataset | [ByteDance/FullStackBench](https://huggingface.co/datasets/ByteDance/FullStackBench) | ## How FullStackBench (English) Is Scored Each task is graded pass@1 via actual code execution against unit tests inside the SandboxFusion sandbox — a submission either passes all associated tests or it doesn't. Because the dataset spans 16 languages and 5+ application domains, results are typically broken down by language and domain rather than reported as a single aggregate score, since performance can vary substantially across both axes. ## State-of-the-Art Results > Scores vary substantially by language and application domain. See the [official GitHub repository](https://github.com/bytedance/FullStackBench) and paper for the full per-model, per-language, per-domain breakdown. ## FullStackBench (English) on Benchgen > No Benchgen results yet — [be the first to run FullStackBench (English)](/benchmarks/bytedance-seed-foundation-code-team/fullstackbench-english). ## FullStackBench (English) vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **FullStackBench (English)** | Multi-domain, multilingual full-stack coding | ≈1,690 | medium | | [HumanEval+](/benchmarks/jiawei-liu-chunqiu-steven-xia-yuyao-wang-lingming-zhang-university-of-illinois-urbana-champaign/humaneval) | Rigorous single-function Python generation | 164 | high | | [RepoBench](/benchmarks/tianyang-liu-canwen-xu-julian-mcauley-uc-san-diego/repobench) | Repository-level code completion | — | medium | FullStackBench is distinguished from narrow single-function benchmarks like HumanEval+ by spanning real-world application domains (data analysis, ML, web/desktop dev) across 16 languages rather than isolated algorithmic Python problems. ## Run FullStackBench (English) on Your Model Benchgen lets teams evaluate their own coding model against FullStackBench's English-instruction tasks across 16 languages and real-world domains, surfacing language- and domain-specific weaknesses a single-language benchmark would miss. ## Frequently Asked Questions

What is FullStackBench?

FullStackBench is a ByteDance Seed benchmark covering full-stack programming across 16 languages and real-world application domains, executed via the SandboxFusion sandbox; this page covers its English-instruction split.

What does a good FullStackBench score look like?

Because performance varies substantially by language and domain, a strong result means maintaining high pass@1 consistently across languages and domains rather than excelling in only one or two.

Who created FullStackBench?

FullStackBench was created by the ByteDance Seed Foundation Code Team, published in late 2024.

Rank	Model	Score
1	o1-preview	66.12
2	o1-mini	61.44
3	claude-3-5-sonnet	60.7
4	deepseek-v2-5	57.2
5	qwen2-5-coder-32b-instruct	52.4
6	qwen2-5-72b-instruct	52.19
7	llama-3-1-70b-instruct	50.65
8	qwen2-5-coder-7b-instruct	48.16

FullStackBench (English) — Results

FullStackBench (English) — Evaluations