VLGuard — Results
| Rank | Model | Score |
|---|---|---|
| 1 | shieldstral-1-0 | 97.7 |
| 2 | omniguard-7b | 88.5 |
| 3 | nemotron-3-5-content-safety-4b | 84.2 |
| 4 | llavaguard-7b | 69.5 |
| 5 | shieldgemma-2-4b | 61.3 |
| 6 | llamaguard-4-12b | 59.9 |
VLGuard
1 phaseActive
A vision-language safety dataset for evaluating multimodal harmful content classification. Zong et al., 2024.
Overview
VLGuard
Quick answer: VLGuard measures whether a classifier correctly identifies harmful content in image-plus-text (vision-language) inputs, spanning categories like unsafe images paired with benign-looking text and safe images paired with harmful instructions. It's the benchmark where Shieldstral posts its strongest score of any comparison in Mistral's published results — 97.7% F1 — reflecting its native multimodal safety design.
At a Glance
What it tests: Multimodal safety classification across image-instruction pairs, including cases where harm arises only from the combination of image and text rather than either modality alone. Why it matters: As multimodal LLMs proliferate, safety classifiers built purely for text miss harm that's only visible when image and text are evaluated jointly — VLGuard was one of the earlier datasets specifically targeting this gap. Known limitations: Safety-relevant image data is inherently harder to source responsibly than text (it can't be freely synthesized), so the dataset's absolute size is more limited than large text-only safety corpora.
What VLGuard Measures
VLGuard pairs images with instructions/queries and labels the combination as safe or unsafe, covering scenarios where an unsafe image is paired with an innocuous-seeming caption, and where a safe image is paired with a harmful instruction that only becomes actionable in the visual context. This directly tests whether a guard model's multimodal reasoning genuinely integrates both modalities rather than defaulting to a text-only or image-only judgment.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Safety / multimodal content moderation |
| Metric | F1 score (%) |
| Modality | Image + text |
| Created by | Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, Timothy Hospedales |
| Paper | Safety Fine-Tuning at (Almost) No Cost / VLGuard (arXiv 2402.02207) |
| GitHub | ys-zong/VLGuard |
How VLGuard Is Scored
Each image-instruction pair's ground-truth safety label is compared against a classifier's prediction and scored as F1.
State-of-the-Art Results
Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%). Only models with native multimodal support report scores here.
| Rank | Model | Score | Source | Date |
|---|---|---|---|---|
| 1 | Shieldstral 1.0 | 97.7% | Mistral AI model card | 2026-08 |
| 2 | OmniGuard-7B | 88.5% | Mistral AI model card | 2026-08 |
| 3 | Nemotron-3.5-Content-Safety-4B | 84.2% | Mistral AI model card | 2026-08 |
| 4 | LlavaGuard-7B | 69.5% | Mistral AI model card | 2026-08 |
| 5 | ShieldGemma-2-4B | 61.3% | Mistral AI model card | 2026-08 |
| 6 | LlamaGuard-4-12B | 59.9% | Mistral AI model card | 2026-08 |
Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements. Text-only guard models (GPT-OSS-Safeguard, Qwen3Guard, ShieldGemma-9B) do not report scores on this multimodal benchmark.
VLGuard on Benchgen
No Benchgen results yet — be the first to run VLGuard.
VLGuard vs Other Benchmarks
| Benchmark | What it tests | Modality | Saturation |
|---|---|---|---|
| VLGuard | Image-instruction pair safety | Image + text | High |
| UnsafeBench | Standalone unsafe image classification | Image only | Medium |
| LlavaGuard | Fine-grained multimodal policy safety | Image + text | Medium |
Run VLGuard on Your Model
Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's VLGuard scores to compare against Shieldstral and other multimodal guard models.
Related Models & Benchmarks
- Models: Shieldstral 1.0 · LlavaGuard-7B · LlamaGuard-4-12B
- Benchmarks: UnsafeBench · LlavaGuard
Frequently Asked Questions
What is VLGuard?
VLGuard is a vision-language safety benchmark pairing images with instructions, testing whether a classifier can detect harm that arises from the combination of image and text.Why do text-only guard models not appear on this leaderboard?
VLGuard requires native image understanding — classifiers without a vision encoder (e.g., most text-only guard models) cannot process this benchmark's inputs.Who created VLGuard?
Yongshuo Zong and colleagues, published in 2024.Last updated 2026-08-12.