# HardMultiQA ![Category](https://img.shields.io/badge/Category-Knowledge-1f6feb) ![Metric](https://img.shields.io/badge/Metric-%25%20Accuracy-2da44e) ![Language](https://img.shields.io/badge/Language-Russian-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) > **Quick answer:** HardMultiQA is a challenging 5-shot benchmark of factual knowledge in Russian, released by Yandex with its evaluation protocol alongside AliceAI-Foundation-80B-A3B-Base. The model scores 67.9, ahead of DeepSeek-V4-Flash-Base (65.4), Nemotron-3-Super-120B-A12B-Base (54.5) and GLM-4.5-Air-Base (48.6) in Yandex's own comparison. ## At a Glance **What it tests:** Harder, multi-fact questions about the Russian-language world, evaluated 5-shot on base models. **Why it matters:** It separates models with deep Russian-language knowledge from those with only shallow coverage, where easier benchmarks saturate. **Known limitations:** Scores are from Yandex's own evaluation infrastructure and compare base models only. ## What HardMultiQA Measures HardMultiQA is the harder companion to [WikiWebFacts](/benchmarks/yandex/wikiwebfacts): questions require retrieving several facts from memory, with no tools or retrieval, in a Russian-language context. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Knowledge (Russian) | | Metric | % accuracy, 5-shot | | Created by | Yandex | | Dataset | [huggingface.co/datasets/yandex/HardMultiQA](https://huggingface.co/datasets/yandex/HardMultiQA) | ## How HardMultiQA Is Scored Models are prompted with five examples and scored on answer accuracy using Yandex's published evaluation protocol. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [AliceAI-Foundation-80B-A3B-Base](/models/yandex/aliceai-foundation-80b-a3b-base) | 67.9 | [Yandex model card](https://huggingface.co/yandex/AliceAI-Foundation-80B-A3B-Base) | 2026-09 | *Yandex's own comparison (not on Benchgen): DeepSeek-V4-Flash-Base 65.4, Nemotron-3-Super-120B-A12B-Base 54.5, GLM-4.5-Air-Base 48.6, Qwen3.5-35B-A3B-Base 47.2.* ## HardMultiQA on Benchgen > No Benchgen results yet — be the first to run HardMultiQA. ## HardMultiQA vs Other Benchmarks | Benchmark | What it tests | Language | |---|---|---| | **HardMultiQA** | Hard multi-fact knowledge, 5-shot | Russian | | [WikiWebFacts](/benchmarks/yandex/wikiwebfacts) | Factual knowledge, 5-shot | Russian | | [TriviaQA](/benchmarks/triviaqa) | Trivia knowledge | English | ## Run HardMultiQA on Your Model Download the dataset and evaluation protocol from the Hugging Face dataset page. ## Related Models & Benchmarks - [AliceAI-Foundation-80B-A3B-Base](/models/yandex/aliceai-foundation-80b-a3b-base) - [WikiWebFacts](/benchmarks/yandex/wikiwebfacts) ## Frequently Asked Questions

What is HardMultiQA?

A challenging 5-shot benchmark from Yandex that measures multi-fact knowledge in Russian.

What is a good HardMultiQA score?

In Yandex's comparison, 67.9 (AliceAI-Foundation-80B-A3B-Base) leads; other strong open base models score about 47 to 65.

Who created HardMultiQA?

Yandex, released openly on Hugging Face with the AliceAI-Foundation-80B-A3B-Base model.

HardMultiQA — Results

HardMultiQA — Phases

HardMultiQA