Research map / Vision-language and multimodal / Multimodal LLMs

THEMIS: Towards Holistic Evaluation of MLLMs for Scientific Paper Fraud Forensics

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces THEMIS, a multi-task benchmark of ~4K questions from retracted-paper cases and synthetic data for evaluating MLLMs on visual scientific-paper fraud forensics, finding that the best model (GPT-5) reaches only 56.15%.

Problem

Existing benchmarks for MLLM visual forensics lack real-world complexity, fine-grained fraud-type diversity, and multi-dimensional capability evaluation for detecting scientific paper fraud.

Approach

A benchmark combining real retracted-paper fraud cases with synthetically generated manipulations (splicing, copy-move, AI-generated, duplication, text-image inconsistency), three task formats, 16 fine-grained manipulation operations, and a mapping from fraud types to five reasoning capabilities, plus a Balanced Robustness Index.

Key results

• GPT-5 achieves the best overall BRI of 56.15%, while Qwen2.5-VL-72B reaches 47.16% among open-source models.
• GPT-5's F1 on Duplication identification drops from 38% to 17% as stacked manipulations increase from one to three.
• GPT-5 scores 53.50% on Single-Mode Forgery identification versus 24.14% on localization.

Surprising findings

• Few-shot prompting gives mixed results, while chain-of-thought consistently helps, with the weaker Llama 4 Maverick benefiting more from CoT than GPT-5.
• Synthetic fraud data is about as deceptive as real retracted-paper fraud data, with splicing being the main exception.

Limitations

• Real-world subset is small, with only 152 cases from 41 retracted papers.
• Synthetic data generated with scripted and generative pipelines may not fully match real fraud distributions.

Experimental setup

Datasets: Retraction Watch / PubPeer retracted papers PubMed Central papers

Benchmarks: THEMIS

Metrics: accuracy IoU set-based F1 text-based F1 Balanced Robustness Index

Baselines: FakeBench DiffuSyn Bench SHIELD MFC-Bench MMFakeBench Forensics-Bench

Builds on: CIMD GIM Segment Anything Model YOLOv7

Closest papers by meaning

• Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models (CVPR 2025 · similarity 0.88)
• BusterX: MLLM-Powered AI-Generated Video Forgery Detection and Explanation (NeurIPS 2026 · similarity 0.86)
• Identity-Aware Vision-Language Model for Explainable Face Forgery Detection (AAAI 2026 · similarity 0.86)
• Locate-Then-Examine: Grounded Region Reasoning Improves Detection of AI-Generated Images (CVPR 2026 · similarity 0.86)
• The Coherence Trap: When MLLM-Crafted Narratives Exploit Manipulated Visual Contexts (CVPR 2026 · similarity 0.85)
• ForgerySleuth: Empowering Multimodal Large Language Models for Image Manipulation Detection (NeurIPS 2025 · similarity 0.85)