OpenRCA
1 phaseActive
Microsoft's benchmark testing whether LLMs can locate the root cause of real enterprise software failures from large-scale telemetry data.
Overview
OpenRCA
Quick answer: OpenRCA is a Microsoft-created benchmark asking "Can Large Language Models Locate the Root Cause of Software Failures?", built from 335 real failures across three enterprise software systems (Telecom, Bank, and Market) with over 68GB of accompanying logs, metrics, and traces that a model must analyze to identify the true root cause.
At a Glance
What it tests: Whether an LLM (or agent) can take a natural-language query plus large volumes of raw telemetry data and correctly diagnose the root cause of a real production software failure.
Why it matters: Root-cause analysis (RCA) is one of the most time-consuming tasks in enterprise software operations; OpenRCA directly tests whether current LLMs can meaningfully accelerate this process using the same messy, large-scale telemetry human engineers rely on.
Known limitations: With 68GB+ of logs, metrics, and traces per system, OpenRCA is heavily constrained by context-window and retrieval limitations, meaning scores reflect an entangled mix of root-cause reasoning ability and an agent's data-retrieval/tool-use strategy.
What OpenRCA Measures
OpenRCA draws on 335 real incidents across three enterprise systems — Telecom, Bank, and Market (the latter two further split into cloudbed-1/cloudbed-2 sub-environments) — each paired with a natural-language query and the full telemetry record (logs, metrics, and distributed traces) surrounding the failure. Given this data, a model must identify the true root cause of the incident, mirroring the diagnostic workflow of a human site-reliability or DevOps engineer.
Because the underlying telemetry for each incident can span gigabytes of raw data, OpenRCA is explicitly designed to test agentic systems' ability to search, filter, and reason over large volumes of operational data rather than answering from a small, pre-filtered context window.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Agent / root cause analysis from enterprise telemetry |
| Metric | Root cause identification accuracy (%) |
| Number of tasks | 335 failures across 3 enterprise systems |
| Saturation | Low — large telemetry volumes and multi-step diagnosis make this a hard, unsaturated task for current LLMs and agents |
| Created by | Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qingwei Lin, Dan Pei, Pinjia He, Dongmei Zhang, Qi Zhang (Microsoft) |
| Source paper | Xu et al. 2025, ICLR |
| GitHub | microsoft/OpenRCA |
How OpenRCA Is Scored
A model's output — its identified root cause for a given incident query — is compared against the ground-truth root cause recorded for that incident; the headline metric is the percentage of the 335 incidents where the model's diagnosis matches the documented true root cause.
State-of-the-Art Results
OpenRCA's authors note the benchmark is used to explore how far agent systems can go in locating real software root causes, with current LLM-based approaches leaving substantial headroom against the large-scale, multi-system telemetry involved. See the official GitHub repository for the current per-model breakdown.
OpenRCA on Benchgen
No Benchgen results yet — be the first to run OpenRCA.
OpenRCA vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| OpenRCA | Root-cause analysis from enterprise telemetry | 335 | low |
| WideSearch | Broad, verifiable web information-gathering | 200 | very low |
| Terminal-Bench Hard | Hard, realistic CLI agent tasks | 89 | low |
OpenRCA is distinguished from other agentic benchmarks by centering on operational telemetry analysis at enterprise scale, rather than web search or terminal command execution, making it a distinct diagnostic-reasoning signal for AIOps-style agents.
Run OpenRCA on Your Model
Benchgen lets teams evaluate their own model against OpenRCA's real enterprise-incident dataset, surfacing whether it can reliably diagnose root causes from large-scale operational telemetry.