Research map / Vision-language and multimodal / Multimodal LLMs

Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Deliberately contaminates LLaVA-v1.5-7B and Qwen2-VL-7B on MMStar/RealWorldQA and proposes detecting contamination by checking whether accuracy drops on multi-modal semantically perturbed benchmark variants (image edited with Flux ControlNet, answer changed).

Problem

Detecting test-set leakage (data contamination) in vision-language models, where existing LLM-oriented detectors fail or behave inconsistently across fine-tuning regimes.

Approach

Generate visually perturbed benchmark items of similar or lower difficulty (GPT-4o caption conditioned on the new answer, rendered by Flux+ControlNet with Canny edges), and flag contamination when a model's accuracy drops on the perturbed set while clean models improve; requires no clean-model reference.

Key results

• Contaminated LLaVA-v1.5-7B and Qwen2-VL-7B show accuracy drops on the perturbed set across all epochs and training strategies (e.g., Qwen2-VL-7B LLM-only epoch 3 on MMStar: -48.08 points), while clean models improve.
• Multi-modal semantic perturbation flags all contaminated models without a clean reference, whereas CircularEval and Choice Confusion miss several contaminated settings (e.g., Choice Confusion misses all LLaVA-v1.5-7B contaminated settings on MMStar).
• On NaturalBench counterfactuals, contaminated LLaVA-v1.5-7B drops by up to 45.58 points; with o3 automated filtering, detection still holds (294 retained pairs).

Surprising findings

• Choice Confusion fails to flag contaminated LLaVA-v1.5-7B on MMStar regardless of training strategy or epoch count.
• Multi-modal leakage (text-only performance) misses LLaVA-v1.5-7B trained 3 epochs with standard fine-tuning, and its signal breaks for some Qwen2-VL-7B settings.

Limitations

• Requires questions with strict visual grounding; method is invalid for questions answerable without the image.
• Manual filtering needed for the main results because diffusion models render text and fine detail imperfectly; about 1.8% to 3.4% of perturbed images deviate from original visual details.
• Evaluated on only two open VLMs (plus LLaVA-13B) and two multiple-choice benchmarks with synthetic contamination via continual fine-tuning.

Experimental setup

Datasets: MMStar RealWorldQA NaturalBench MathVista MMMU MMBench CV-Bench LLaVA-v1.5 665K instruction data

Benchmarks: MMStar RealWorldQA NaturalBench

Metrics: accuracy accuracy drop (delta) between original and perturbed sets text-only accuracy circular-eval accuracy choice confusion accuracy

Baselines: Multi-modal Leakage CircularEval Choice Confusion N-gram Accuracy Shared Likelihood Guided Prompting

Builds on: LoRA LLaMA-Factory ControlNet Multi-modal Leakage detection CircularEval Choice Confusion

Closest papers by meaning

• ConStat: Performance-Based Contamination Detection in Large Language Models (NeurIPS 2024 · similarity 0.87)
• VLMs can Aggregate Scattered Training Patches (NeurIPS 2025 · similarity 0.87)
• Time Travel in LLMs: Tracing Data Contamination in Large Language Models (ICLR 2024 · similarity 0.87)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.87)
• Task Contamination: Language Models May Not Be Few-Shot Anymore (AAAI 2024 · similarity 0.87)
• Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? (CVPR 2025 · similarity 0.86)