MusicCaps
1 phaseActive
5,521 music clips with rich expert text descriptions, the standard evaluation set for text-to-music generation models like MusicLM.
Overview
MusicCaps
Quick answer: MusicCaps is a 5,521-clip dataset of rich, human-expert-written music captions released alongside Google's MusicLM paper, and it has become the de facto standard evaluation set for text-to-music generation systems.
At a Glance
What it tests: Whether a text-to-music generation model produces audio that faithfully matches a detailed text description (e.g., "a calming violin melody backed by a distorted guitar riff") while maintaining audio fidelity and coherence over time.
Why it matters: Before MusicCaps, evaluating text-to-music systems lacked a shared, richly-annotated reference set; its expert-written captions cover both objective musical attributes and subjective qualities, enabling consistent comparison across models like MusicLM, MusicGen, and successors.
Known limitations: MusicCaps is a fixed 10-second-clip evaluation set, so it primarily measures short-form generation quality rather than long-form musical coherence, and automated metrics (FAD, CLAP score) are still imperfect proxies for genuine musical quality.
What MusicCaps Measures
MusicCaps consists of 5,521 music clips, each paired with a rich text description written by human music experts, covering genre, mood, instrumentation, and other musical attributes in natural language. It was released as supporting material for MusicLM, a hierarchical sequence-to-sequence text-to-music generation model that produces 24kHz audio consistent over several minutes.
Because captions are written by experts rather than crowd-workers, MusicCaps descriptions tend to be more musically precise than typical caption datasets, making it a stronger signal for evaluating whether generated audio matches nuanced textual instructions (e.g., matching both instrumentation and mood, not just genre).
MusicCaps is most commonly used in two directions: as ground truth for computing similarity-based generation metrics (Fréchet Audio Distance, CLAP score) between generated and reference audio, and as a source of text prompts to drive text-to-music generation for human or automated evaluation.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Multimodal (text-to-music generation evaluation) |
| Metric | Fréchet Audio Distance (FAD), CLAP score, and/or Mean Opinion Score (MOS) depending on evaluation setup |
| Number of tasks | 5,521 music clips with expert text captions |
| Saturation | Medium — generation quality has improved substantially since MusicLM but nuanced adherence remains imperfect |
| Created by | Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, and collaborators |
| Source paper | Agostinelli et al. 2023 |
| Dataset | Hugging Face |
How MusicCaps Is Scored
Evaluation typically combines an automated audio-similarity metric (Fréchet Audio Distance between generated and reference clips) with a text-audio adherence metric (e.g., CLAP score, measuring how well the generated audio matches the caption), and often human Mean Opinion Score ratings for overall quality and adherence.
State-of-the-Art Results
Scores vary heavily by generation architecture and evaluation protocol (FAD variant, CLAP model version). See published technical reports for current model-by-model comparisons.
MusicCaps on Benchgen
No Benchgen results yet — be the first to run MusicCaps.
MusicCaps vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| MusicCaps | Text-to-music generation quality & caption adherence | 5,521 clips | medium |
| MME | Multimodal LLM perception + cognition | 2,374 | high |
| MTVQA | Multilingual text-centric visual QA | 6,778 | low |
MusicCaps is unique among Benchgen's multimodal benchmarks in evaluating audio generation rather than vision-language understanding, making it a key reference for teams building or fine-tuning music-generation models.
Run MusicCaps on Your Model
Benchgen lets teams evaluate their own text-to-music models against MusicCaps prompts on demand, tracking audio-similarity and caption-adherence metrics over time as generation architectures evolve.
Related Models & Benchmarks
Explore related multimodal benchmarks like MME and MTVQA to compare vision-language evaluation with MusicCaps' audio-generation focus.