Research map / Vision-language and multimodal / Visual question answering
VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VDC, a training-free detector that flags dirty samples (poisoned images and noisy labels) by asking an MLLM (InstructBLIP) questions generated from the label and scoring answer consistency, outperforming poisoned-sample and noisy-label baselines on CIFAR-10 and ImageNet subsets.
Problem
Detecting dirty training samples (backdoor-poisoned images, noisy labels, and their hybrids) generalizes poorly across dirty-sample categories and attack/noise types in existing detectors.
Approach
Exploit visual-linguistic inconsistency between an image and its label: an LLM generates general and label-specific visual questions, an MLLM answers them on the image, and an answer-evaluation step (string matching or ChatGPT) yields a matching score; samples below a threshold are removed.
Key results
- On CIFAR-10 with eta=0.09 poisoning, VDC averages 99.91% TPR and 2.75% FPR across six backdoor attacks, versus 97.94% TPR / 0.48% FPR for the best baseline CT and 77.20% TPR for SPECTRE.
- On asymmetric noisy labels on CIFAR-10 (eta=0.4), VDC achieves 99.60% TPR versus 59.67% for SimiFeat-V.
- On the hybrid poisoned plus noisy CIFAR-10 setting, VDC averages 99.41% TPR while SCAn drops from 95.46% to 46.29%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VDC instead of CT for detecting poisoned samples | CIFAR-10 · CIFAR-10, poisoned samples, η=0.09, average across triggers | TPR (%): 97.94 -> 99.91 | Helped |
| Use VDC instead of the best baseline TPR, STRIP | ImageNet-100 poisoned sample detection · ImageNet-100, poisoned samples, η=0.099, average | TPR (%): 82.37 -> 99.94 | Helped |
| Use VDC instead of the strongest baseline TPR, SPECTRE | ImageNet-Dog poisoned sample detection · ImageNet-Dog, poisoned samples, η=0.09, average | TPR (%): 87.43 -> 98.68 | Helped |
| Use VDC instead of SimiFeat-R | CIFAR-10 symmetric noisy-label detection · CIFAR-10, symmetric noisy labels, η=0.4 | TPR (%): 99.16 -> 98.81 | Mixed |
| Use VDC instead of BHN | CIFAR-10 asymmetric noisy-label detection · CIFAR-10, asymmetric noisy labels, η=0.4 | TPR (%): 83.13 -> 99.60 | Helped |
| Use VDC for detecting both poisoned samples and noisy labels | CIFAR-10 hybrid dirty sample detection · CIFAR-10, hybrid poisoned samples and noisy labels | TPR (%): 75.08 -> 99.41 | Helped |
| Use VDC to filter poisoned samples before retraining instead of no detection | ResNet-18 trained on purified CIFAR-10 · CIFAR-10 poisoned samples, η=0.09, average across triggers | ASR (%): 98.00 -> 1.63 | Helped |
| Use only general questions rather than both general and label-specific questions (ablation) | VDC · CIFAR-10 poisoned sample detection | Detection performance | Hurt |
| Use only label-specific questions rather than both question types (ablation) | VDC · CIFAR-10 poisoned sample detection | Detection performance | Hurt |
| Add label-specific questions alongside general questions (ablation) | VDC · CIFAR-10 poisoned sample detection | Detection performance | Helped |
| Increase the number of visual questions (ablation) | VDC · ImageNet-100 poisoned sample detection | Detection performance | Helped |
| Replace InstructBLIP with Otter (ablation) | VDC · CIFAR-10 poisoned sample detection | TPR (%) | Hurt |
Surprising findings
- A naive CLIP-based matching score (VDC-CLIP) reaches only about 28-52% TPR on CIFAR-10, far below the full MLLM pipeline.
Limitations
- Cannot detect clean-label backdoor attacks since it relies on label-image inconsistency.
- LLM and MLLM can produce incorrect answers despite ensembling.
- Inference cost grows with the number of questions.
Experimental setup
Datasets: CIFAR-10 ImageNet-100 ImageNet-Dog
Benchmarks: CIFAR-10 ImageNet-100 ImageNet-Dog
Metrics: TPR FPR ASR ACC
Baselines: STRIP SS SCAn Frequency CT D-BR SPECTRE BHN CL CORES SimiFeat-V SimiFeat-R
Builds on: InstructBLIP ChatGPT CLIP
Closest papers by meaning
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples (NeurIPS 2024 · similarity 0.88)
- MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questions (AAAI 2026 · similarity 0.86)
- VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning (CVPR 2024 · similarity 0.86)
- VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models (AAAI 2024 · similarity 0.86)
- FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.86)
- Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.