Research map / Vision-language and multimodal / Multimodal LLMs
SALART-VQA: Diagnosing Whether VLMs Understand Salient Artifacts in Generated Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SalArt-VQA, a closed-set multiple-choice VQA benchmark (950 images, 3,681 questions) that diagnoses whether VLMs detect, localize, ground, and justify salient artifacts in AI-generated images, and evaluates 20 VLMs.
Problem
Image-level artifact detection accuracy hides whether VLMs actually localize, ground, and describe the defect they flag, and whether they abstain when the defect is absent.
Approach
A four-question aligned MCQ chain (presence, semantic region, bounding box, evidence description) with matched real and paired generated reference images, scored with all-correct image-level metrics to expose a sensitivity-calibration tradeoff.
Key results
- Gemini 3.1 Pro reaches 99.37% Q1 detection recall on artifact images but only 53.26% All Q1-4 accuracy on artifact images.
- Claude 4.6 Opus, GPT-5.4 and Gemini-style sensitive models show a sensitivity-calibration tradeoff; GLM-4.6V gets 97.47% Ref-Real All Q1-4 but 0.21% artifact All Q1-4.
- Across 20 VLMs, no model combines high artifact sensitivity with strong reference-side calibration; the best overall All Q1-4 is 48.01% (Claude 4.6 Opus).
Surprising findings
- The strongest artifact-detection model (Gemini 3.1 Pro, 99.37% recall) has 0% Ref-PG All Q2-4 accuracy and only 7.87% Ref-Real complete accuracy.
- Models that are well calibrated on reference images (e.g., GLM-4.6V) achieve this largely by avoiding artifact commitments, scoring near random on artifact images.
Limitations
- Benchmark is small (950 images), with artifact images limited to simple compositions and salient, describable defects.
- Inpainting-based Art-PG split relies on a specific pipeline (MediaPipe, SAM 2, FLUX.1-Fill-dev) and generator mix may not represent current generators fully.
- Task-adapted localization baselines require forced conversion to box choices, which limits their fair comparison.
Experimental setup
Datasets: COCO captions COCO photographs
Benchmarks: SalArt-VQA
Metrics: Q1 detection recall Q1 specificity All Q1-4 complete accuracy All Q2-4 grounding accuracy
Baselines: PAL4VST LEGION SIDA-7B Random Human
Builds on: POPE HallusionBench LEGION PAL4VST SIDA
Closest papers by meaning
- Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanation (NeurIPS 2025 · similarity 0.86)
- Locate-Then-Examine: Grounded Region Reasoning Improves Detection of AI-Generated Images (CVPR 2026 · similarity 0.86)
- Identity-Aware Vision-Language Model for Explainable Face Forgery Detection (AAAI 2026 · similarity 0.85)
- Semi-Truths: A Large-Scale Dataset of AI-Augmented Images for Evaluating Robustness of AI-Generated Image detectors (NeurIPS 2024 · similarity 0.85)
- VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.85)
- BIAS-ID: A Framework for Analyzing Transformation Biases in AI-Generated Image Detectors (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.