Research map / Vision-language and multimodal / Multimodal LLMs

Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds a 15K-concept multimodal dictionary (DACO-400K) from WordNet and CC-3M, uses it to initialize and annotate TopK/L1 sparse autoencoders, and steers frozen MLLMs at inference to improve jailbreak/unsafe-query safety while keeping utility.

Problem

MLLMs can produce unsafe responses to malicious text-image queries; existing activation steering methods handle few safety concepts, struggle to calibrate steering strength, and SAE features lack semantic grounding.

Approach

Use a large CLIP-retrieved multimodal concept dictionary both for training-free oblique-projection sparse coding and to initialize SAE decoder atoms, then annotate atoms as desirable or undesirable by cosine distance to expert-labeled concept centroids, and steer by suppressing undesirable and amplifying desirable coefficients.

Key results

• On Qwen2.5-VL-7B, DACO raises MS-R from 0.442 (no steering) to 0.990 and JBV-QG from 0.543 to 0.841 while MMMU drops only from 0.546 to 0.521.
• On InternVL3.5-8B, DACO reaches JBV-QG 0.970 versus 0.926 for MOP and 0.847 for ActAdd.
• Initializing TopK-SAE with the concept dictionary raises JBV-QG from 0.822 to 0.841 and MMMU from 0.505 to 0.521.

Surprising findings

• Using text-only concept stimuli instead of multimodal retrieval degrades JBV-R from 0.903 to 0.848, showing multimodal retrieval matters.
• ActAdd over-refuses benign MOSSBench queries, with average over-refusal 0.18 versus 0.09 for no steering.

Limitations

• Steering strength and annotation threshold require tuning (eta, gamma), with trade-offs between safety and MMMU.
• Concept labels rely on an expert MLLM (Qwen3-VL-32B) to partition concepts as desirable or undesirable.
• Intervention covers only the first 50 generated tokens and selected layers 15-19.

Experimental setup

Datasets: WordNet CC-3M JailBreakV-28K MM-SafetyBench MOSSBench MMMU

Benchmarks: MM-SafetyBench JailBreakV-28K MMMU MOSSBench MM-Vet MM-Vet v2

Metrics: defense success rate fluency perplexity MMMU accuracy over-refusal rate FVE DA DO FDN neighborhood purity

Baselines: No Steering Prompting (MM-SafetyBench safety prompt) ActAdd MOP Khayatan et al. steering

Builds on: Parsimonious Concept Engineering (PaCE) Activation Addition TopK sparse autoencoder CLIP LoRA

Closest papers by meaning

• MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.90)
• Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? (CVPR 2025 · similarity 0.90)
• Pragma-VL: Towards a Pragmatic Arbitration of Safety and Helpfulness in MLLMs (ICLR 2026 · similarity 0.88)
• GuardAlign: Robust Safety Alignment in Multimodal Large Language Models (ICLR 2026 · similarity 0.88)
• Multimodal Situational Safety (ICLR 2025 · similarity 0.88)
• SDEval: Safety Dynamic Evaluation for Multimodal Large Language Models (AAAI 2026 · similarity 0.88)