Research map / Vision-language and multimodal / Multimodal LLMs
HEIE: MLLM-Based Hierarchical Explainable AIGC Image Implausibility Evaluator
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HEIE, an InternVL-8B-based MLLM that jointly outputs hierarchical (local+global) defect heatmaps, a verisimilitude score, and CoT-driven explanations for AIGC image implausibility, and releases the Expl-AIGI-Eval dataset.
Problem
Specialized small models predict AIGC implausibility heatmaps but lack explainability and common-sense generalization, while MLLMs struggle with fine-grained defect localization and cannot output pixel-level heatmaps.
Approach
Adds a [MAP] token whose LLM hidden state drives a two-way cross-attention heatmap mapper over ViT features, combines local (per-patch) and global heatmaps with uncertainty-based Laplace weighting, and uses a [SCORE] token plus heatmap-derived score calibrated into a verisimilitude score, all within a five-step CoT that also generates explanations.
Key results
- On RichHF-18K, HEIE reaches MSE 0.00825 and CC 0.574 for heatmaps and PLCC 0.697 / SRCC 0.683 for scores, versus RAHF (augmented prompt) at 0.00920 MSE, 0.556 CC, 0.693 PLCC.
- On Expl-AIGI-Eval, HEIE gets GPT-4o eval 4.58 and human eval 4.35, above GPT-4o (3.83 / 4.00) and Claude-3.5-Sonnet (3.94 / 4.08).
- Zero-shot cross-domain: trained on RichHF-18K and tested on AbHuman, HEIE achieves MSE 0.00858 vs. 0.01170 for EVA-CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HEIE instead of the strongest reported baseline | RichHF-18K heatmap prediction and plausibility scoring · RichHF-18K | MSE (All Data), CC, PLCC, SRCC: 0.00920 -> 0.00825; 0.556 -> 0.574; 0.693 -> 0.697; 0.681 -> 0.683 | Helped |
| Use HEIE instead of the strongest baseline | AbHuman heatmap prediction · AbHuman | MSE (All Data), KLD, CC, SIM, AUC-Judd: 0.00916 -> 0.00510; 1.953 -> 1.629; 0.582 -> 0.684; 0.154 -> 0.423; 0.636 -> 0.938 | Helped |
| Use HEIE to generate explanations instead of baseline MLLMs | Expl-AIGI-Eval explanation generation · Expl-AIGI-Eval | GPT-4o Eval and Human Eval: 3.938220 -> 4.582199; 4.080628 -> 4.352880 | Helped |
| Use only local rather than only global map tokens/features (ablation) | Adaptive Hierarchical Implausibility Mapper · RichHF-18K | MSE: 0.01071 -> 0.00980 | Helped |
| Combine global and local tokens with fixed weights (ablation) | Adaptive Hierarchical Implausibility Mapper · RichHF-18K | MSE: 0.00980 -> 0.00954 | Helped |
| Use learnable weights instead of fixed weights (ablation) | Adaptive Hierarchical Implausibility Mapper · RichHF-18K | MSE: 0.00954 -> 0.00873 | Helped |
| Use uncertainty-based adaptive fusion instead of learnable weights (ablation) | Adaptive Hierarchical Implausibility Mapper · RichHF-18K | MSE: 0.00873 -> 0.00825 | Helped |
| Use the [SCORE] token instead of plain-text LLM scoring (ablation) | Verisimilitude Scorer · RichHF-18K | PLCC: 0.216 -> 0.653 | Helped |
| Predict scores from heatmap data instead of plain-text LLM scoring (ablation) | Verisimilitude Scorer · RichHF-18K | PLCC: 0.216 -> 0.597 | Helped |
| Combine heatmap and [SCORE] token features with calibration (ablation) | Verisimilitude Scorer · RichHF-18K | PLCC: 0.653 -> 0.697 | Helped |
| Use ground-truth CoT text instead of generated CoT text (ablation) | CoT-Driven Explainable System · RichHF-18K | MSE: 0.00825 -> 0.00792 | Helped |
| Add CoT text rather than omit CoT text (ablation) | CoT-Driven Explainable System · RichHF-18K | MSE: 0.00913 -> 0.00825 | Helped |
Surprising findings
- Ablation on the scorer shows plain-text LLM score output has very low accuracy (PLCC 0.216), attributed to LLM numerical insensitivity.
- Ground-truth heatmap plus [SCORE] token gives only modest gains over the predicted heatmap (PLCC 0.743 vs 0.697), suggesting heatmap accuracy matters but is not the sole bottleneck.
Limitations
- Relies on MLLM reasoning efficiency, which the authors identify as an area to improve.
- Explanation annotations are generated by Claude-3.5-Sonnet and GPT-4o rather than human-written.
- RAHF is closed-source, so its comparison results are copied from the paper rather than rerun.
Experimental setup
Datasets: RichHF-18K AbHuman
Benchmarks: RichHF-18K AbHuman Expl-AIGI-Eval
Metrics: MSE KLD CC SIM AUC-Judd PLCC SRCC perplexity GPT-4o score human evaluation score
Baselines: RAHF PickScore EVA-CLIP CLIP InternViT-300M Qwen2-VL-7B-Instruct DeepSeek-VL-7B-chat InternVL2-8B GLM-4V-9B GPT-4o Claude-3.5-Sonnet
Builds on: InternVL RAHF RichHF-18K focal loss Laplace uncertainty
Closest papers by meaning
- SALART-VQA: Diagnosing Whether VLMs Understand Salient Artifacts in Generated Images (NeurIPS 2026 · similarity 0.83)
- TerraVis: Towards Evaluation of World-Grounded Visual Consistency in Text-to-Image Generation via MLLM Workflows (NeurIPS 2026 · similarity 0.83)
- SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories (CVPR 2025 · similarity 0.83)
- DD-CAM: Minimal Sufficient Explanations for Vision Models Using Delta Debugging (NeurIPS 2026 · similarity 0.83)
- UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits (CVPR 2026 · similarity 0.82)
- Image Understanding Makes for A Good Tokenizer for Image Generation (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.