# VLGuard ![Category](https://img.shields.io/badge/Category-Multimodal%20Safety-1f6feb) ![Metric](https://img.shields.io/badge/Metric-F1%20Score-2da44e) ![Saturation](https://img.shields.io/badge/Saturation-High-d29922) [![Paper](https://img.shields.io/badge/Paper-arXiv%202402.02207-b31b1b)](https://arxiv.org/abs/2402.02207) [![GitHub](https://img.shields.io/badge/GitHub-ys--zong%2FVLGuard-181717?logo=github&logoColor=white)](https://github.com/ys-zong/VLGuard) > **Quick answer:** VLGuard measures whether a classifier correctly identifies harmful content in image-plus-text (vision-language) inputs, spanning categories like unsafe images paired with benign-looking text and safe images paired with harmful instructions. It's the benchmark where Shieldstral posts its strongest score of any comparison in Mistral's published results — 97.7% F1 — reflecting its native multimodal safety design. ## At a Glance **What it tests:** Multimodal safety classification across image-instruction pairs, including cases where harm arises only from the combination of image and text rather than either modality alone. **Why it matters:** As multimodal LLMs proliferate, safety classifiers built purely for text miss harm that's only visible when image and text are evaluated jointly — VLGuard was one of the earlier datasets specifically targeting this gap. **Known limitations:** Safety-relevant image data is inherently harder to source responsibly than text (it can't be freely synthesized), so the dataset's absolute size is more limited than large text-only safety corpora. ## What VLGuard Measures VLGuard pairs images with instructions/queries and labels the combination as safe or unsafe, covering scenarios where an unsafe image is paired with an innocuous-seeming caption, and where a safe image is paired with a harmful instruction that only becomes actionable in the visual context. This directly tests whether a guard model's multimodal reasoning genuinely integrates both modalities rather than defaulting to a text-only or image-only judgment. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Safety / multimodal content moderation | | Metric | F1 score (%) | | Modality | Image + text | | Created by | Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, Timothy Hospedales | | Paper | [Safety Fine-Tuning at (Almost) No Cost / VLGuard (arXiv 2402.02207)](https://arxiv.org/abs/2402.02207) | | GitHub | [ys-zong/VLGuard](https://github.com/ys-zong/VLGuard) | ## How VLGuard Is Scored Each image-instruction pair's ground-truth safety label is compared against a classifier's prediction and scored as F1. ## State-of-the-Art Results Scores from the Shieldstral model card (Mistral AI, August 2026). F1 (%). Only models with native multimodal support report scores here. | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) | **97.7%** | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 2 | [OmniGuard-7B](/models/independent-research/omniguard-7b) | 88.5% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 3 | [Nemotron-3.5-Content-Safety-4B](/models/nvidia/nemotron-3-5-content-safety-4b) | 84.2% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 4 | [LlavaGuard-7B](/models/tu-darmstadt-hessian-ai/llavaguard-7b) | 69.5% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 5 | [ShieldGemma-2-4B](/models/google/shieldgemma-2-4b) | 61.3% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | | 6 | [LlamaGuard-4-12B](/models/meta/llamaguard-4-12b) | 59.9% | [Mistral AI model card](https://huggingface.co/mistralai/Shieldstral-1.0-3B) | 2026-08 | *Scores sourced from Mistral AI's published model card, shown for context. Not Benchgen measurements. Text-only guard models (GPT-OSS-Safeguard, Qwen3Guard, ShieldGemma-9B) do not report scores on this multimodal benchmark.* ## VLGuard on Benchgen > No Benchgen results yet — [be the first to run VLGuard](/benchmarks/independent-research/vlguard). ## VLGuard vs Other Benchmarks | Benchmark | What it tests | Modality | Saturation | |---|---|---|---| | **VLGuard** | Image-instruction pair safety | Image + text | High | | UnsafeBench | Standalone unsafe image classification | Image only | Medium | | LlavaGuard | Fine-grained multimodal policy safety | Image + text | Medium | ## Run VLGuard on Your Model Benchgen tracks version-controlled, regression-tested guard-model performance — submit your classifier's VLGuard scores to compare against Shieldstral and other multimodal guard models. ## Related Models & Benchmarks - Models: [Shieldstral 1.0](/models/mistral-ai/shieldstral-1-0) · [LlavaGuard-7B](/models/tu-darmstadt-hessian-ai/llavaguard-7b) · [LlamaGuard-4-12B](/models/meta/llamaguard-4-12b) - Benchmarks: [UnsafeBench](/benchmarks/independent-research/unsafebench) · [LlavaGuard](/benchmarks/tu-darmstadt-hessian-ai/llavaguard) ## Frequently Asked Questions

What is VLGuard?

VLGuard is a vision-language safety benchmark pairing images with instructions, testing whether a classifier can detect harm that arises from the combination of image and text.

Why do text-only guard models not appear on this leaderboard?

VLGuard requires native image understanding — classifiers without a vision encoder (e.g., most text-only guard models) cannot process this benchmark's inputs.

Who created VLGuard?

Yongshuo Zong and colleagues, published in 2024.

--- *Last updated 2026-08-12.*

Rank	Model	Score
1	shieldstral-1-0	97.7
2	omniguard-7b	88.5
3	nemotron-3-5-content-safety-4b	84.2
4	llavaguard-7b	69.5
5	shieldgemma-2-4b	61.3
6	llamaguard-4-12b	59.9

VLGuard — Results

VLGuard — Evaluations