Research map / Vision-language and multimodal / Cross-modal learning
Beyond Text: Fine-Grained Multi-Modal Fact Verification with Hypergraph Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HGTMFC, a hypergraph transformer with line-graph propagation that jointly models claim tokens, text evidence tokens and image patches for fine-grained multi-modal fact verification on MOCHEG.
Problem
Multi-modal fact verification methods mainly capture superficial claim-evidence correlations and miss fine-grained, higher-order cross-modal interactions between claims, text evidence and image evidence.
Approach
Model claim, text evidence and image evidence as a heterogeneous hypergraph over CLIP token/patch nodes, with a trainable group-attention transformer for intra-hyperedge aggregation and heterogeneous attention for node-hyperedge propagation, plus a line-graph GCN branch over hyperedges; the two graph readouts are concatenated for 3-way classification.
Key results
- On MOCHEG with retrieved evidence, HGTMFC reaches 48.61% label accuracy vs 45.62% for MOCHEG.
- With gold evidence, HGTMFC reaches 54.05% label accuracy vs 52.01% for MOCHEG.
- Ablation: replacing heterogeneous attention with homogeneous attention drops gold-evidence LA from 54.05 to 51.02 (mean aggregation 50.66).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add group-attention intra-hyperedge aggregation instead of mean aggregation (ablation) | HGTMFC · MOCHEG retrieval evidence | LA: 46.06 -> 48.61 | Helped |
| Add group-attention intra-hyperedge aggregation instead of mean aggregation (ablation) | HGTMFC · MOCHEG gold evidence | LA: 50.66 -> 54.05 | Helped |
| Use group attention instead of full self-attention (ablation) | HGTMFC · MOCHEG retrieval evidence | LA: 48.12 -> 48.61 | Helped |
| Use group attention instead of full self-attention (ablation) | HGTMFC · MOCHEG gold evidence | LA: 53.36 -> 54.05 | Helped |
| Use heterogeneous attention instead of homogeneous attention (ablation) | HGTMFC · MOCHEG retrieval evidence | LA: 46.19 -> 48.61 | Helped |
| Use heterogeneous attention instead of homogeneous attention (ablation) | HGTMFC · MOCHEG gold evidence | LA: 51.02 -> 54.05 | Helped |
| Add line-graph information propagation (ablation) | HGTMFC · MOCHEG retrieval evidence | LA: 48.16 -> 48.61 | Helped |
| Add line-graph information propagation (ablation) | HGTMFC · MOCHEG gold evidence | LA: 53.48 -> 54.05 | Helped |
Surprising findings
- The text-only variant HGTMFC-w text (50.74% gold LA) already outperforms the full multi-modal MOCHEG baseline on retrieved evidence in some comparisons and is competitive with multi-modal baselines, suggesting limited gain from visual evidence on this benchmark.
Limitations
- Evaluated on a single dataset (MOCHEG) due to limited multi-modal fact-checking datasets.
- Longer training and inference time than CLIP-based baselines.
- Only text and image modalities are implemented; audio and video are claimed extendable but not tested.
Experimental setup
Datasets: MOCHEG
Benchmarks: MOCHEG
Metrics: label accuracy precision recall F1
Baselines: GEAR KGAT HESM Triple-Check MOCHEG Ino MOCHEG-w text Triple-Check-w text
Builds on: HGNN CLIP SBERT Graph Convolutional Network MOCHEG
Closest papers by meaning
- Heterogeneous Graph Reasoning for Fact Checking over Texts and Tables (AAAI 2024 · similarity 0.89)
- Enhancing Multi-Hop Fact Verification with Structured Knowledge-Augmented Large Language Models (AAAI 2025 · similarity 0.87)
- MRR-FV: Unlocking Complex Fact Verification with Multi-Hop Retrieval and Reasoning (AAAI 2025 · similarity 0.87)
- DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts (ICML 2025 · similarity 0.86)
- BiDeV: Bilateral Defusing Verification for Complex Claim Fact-Checking (AAAI 2025 · similarity 0.85)
- Unsupervised Pretraining for Fact Verification by Language Model Distillation (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.