Research map / Vision-language and multimodal / Multimodal LLMs
Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Forensics-Bench, a 63K multi-choice visual question benchmark spanning 112 forgery detection types across five perspectives, and evaluates 22 open-source LVLMs plus GPT-4o, Gemini 1.5 Pro and Claude 3.5 Sonnet.
Problem
Existing benchmarks assess LVLMs' forgery detection only on narrow slices of synthetic media (limited modalities, types, or generators), so no comprehensive evaluation exists of LVLMs' ability to detect diverse forgeries.
Approach
A hierarchical benchmark that organizes forgery data along five perspectives (semantics, modalities, tasks, types, generative models), built by aggregating public forgery datasets' test/validation splits and converting their metadata into multiple-choice QA, with additional robustness and attribution protocols.
Key results
- Best model LLaVA-NEXT-34B reaches 66.7% overall accuracy on Forensics-Bench, with GPT-4o at 57.9%, Gemini 1.5 Pro at 48.3%, and Claude 3.5 Sonnet at 33.8%.
- Forgery attribution accuracy drops sharply: the best model (LLaVA-NEXT-34B) reaches only 44.0% overall, versus strong binary classification performance.
- Under robustness perturbations, color contrast and saturation changes hurt LVLM detection most (e.g., LLaVA-NEXT-34B 62.3% and 61.2% vs. its clean-set score).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate LVLMs on comprehensive forgery detection rather than a narrower task | State-of-the-art LVLMs · Forensics-Bench | overall accuracy | Hurt |
| Use proprietary LVLMs instead of leading open-source LVLMs | Forgery detection · Forensics-Bench | accuracy | Hurt |
| Evaluate forgery binary classification instead of spatial or temporal localization | LVLMs · Forensics-Bench forgery tasks | accuracy | Helped |
| Use local block-wise distortion perturbations | LVLMs · Forensics-Bench robust forgery detection | overall score | Mixed |
| Apply color contrast or saturation perturbations | LVLMs · Forensics-Bench robust forgery detection | overall score | Hurt |
| Ask LVLMs to identify the specific forgery model rather than perform binary forgery classification | LLaVA-NEXT-34B, LLaVA-v1.5-7B-XTuner, InternVL-Chat-V1-2, and mPLUG-Owl2 · Forensics-Bench forgery attribution | accuracy | Hurt |
| Evaluate forgeries from diffusion models rather than GANs | Current LVLMs · Forensics-Bench forgery models | accuracy | Helped |
Surprising findings
- Proprietary models GPT-4o, Gemini 1.5 Pro and Claude 3.5 Sonnet underperform many open-source LLaVA variants, which the authors attribute to conservative answers.
- Local block-wise distortion had a small or even beneficial effect on detection, which the authors attribute to reduced content bias.
Limitations
- Benchmark construction relies on public forgery datasets with heterogeneous sources and generators, and some forgery types map to few datasets.
- Evaluation answer extraction uses ChatGPT as a fallback, which may introduce matching errors.
- Benchmark covers only multiple-choice accuracy, not free-form explanation quality.
Experimental setup
Datasets: HiFi-IFDL DFFD ForgeryNet ForgeryNIR Fake2M GenImage DiffusionForensics CNN-spot FF++ CelebDF-v2 DeeperForensics FFIW
Benchmarks: Forensics-Bench
Metrics: accuracy
Baselines: GPT-4o Gemini 1.5 Pro Claude 3.5 Sonnet LLaVA-NEXT-34B InternVL-Chat-V1-5 mPLUG-Owl2 FakeBench MMFakeBench MFC-Bench
Builds on: MMBench MMMU OpenCompass ForgeryNet DGM4
Closest papers by meaning
- Identity-Aware Vision-Language Model for Explainable Face Forgery Detection (AAAI 2026 · similarity 0.89)
- OmniVL-Guard: Towards Unified Vision-Language Forgery Detection and Grounding via Balanced RL (ICML 2026 · similarity 0.88)
- Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanation (NeurIPS 2025 · similarity 0.88)
- BusterX: MLLM-Powered AI-Generated Video Forgery Detection and Explanation (NeurIPS 2026 · similarity 0.88)
- ForensicHub: A Unified Benchmark & Codebase for All-Domain Fake Image Detection and Localization (NeurIPS 2025 · similarity 0.88)
- THEMIS: Towards Holistic Evaluation of MLLMs for Scientific Paper Fraud Forensics (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.