Research map / Vision-language and multimodal / Cross-modal learning
Frequency Spectrum Is More Effective for Multimodal Representation and Fusion: A Multimodal Spectrum Rumor Detector
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FSRU, a frequency-domain multimodal rumor detector that applies DFT to text and image features, compresses and co-selects spectrum components across modalities, and trains with dual contrastive losses.
Problem
Multimodal (text+image) rumor detection on social media, where existing token-mixing and co-attention fusion give less discriminative unimodal representations and costly, location-dependent cross-modal fusion.
Approach
Move text and image features into the frequency domain via DFT, apply learnable filter banks (unimodal spectrum compression) and cross-modal spectrum co-selection with emphasize/suppress filters, then invert back; fuse modalities via JS-divergence-weighted combination and train with supervised intra-modal and self-supervised inter-modal contrastive losses.
Key results
- FSRU reaches 0.952 accuracy on Twitter and 0.901 on Weibo, versus the best baseline accuracies of 0.911 (LogicDM) on Twitter and 0.892 (SpotFake) on Weibo as listed in the paper's table.
- Ablation: removing unimodal spectrum compression drops Weibo accuracy from 0.901 to 0.866 and Twitter from 0.952 to 0.910.
- FSRU has 1.13M parameters and 9.05G FLOPs, versus 94.39M parameters for BMR and 124.37M for SpotFake (figures from baseline papers).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FSRU instead of the strongest baseline, SpotFake | Multimodal rumor detection · Weibo | Accuracy: 0.892 -> 0.901 | Helped |
| use FSRU instead of the strongest baseline, SpotFake | Multimodal rumor detection · Weibo | Rumor F1: 0.932 -> 0.906 | Hurt |
| use FSRU instead of the strongest baseline, LogicDM | Multimodal rumor detection · Twitter | Accuracy: 0.911 -> 0.952 | Helped |
| use FSRU instead of HMCAN | Multimodal rumor detection · Twitter | Rumor F1: 0.878 -> 0.960 | Helped |
| add unimodal spectrum compression (USC) (ablation) | FSRU · Weibo | Accuracy: 0.866 -> 0.901 | Helped |
| add cross-modal spectrum co-selection (CSC) (ablation) | FSRU · Weibo | Accuracy: 0.883 -> 0.901 | Helped |
| add distribution similarity-based fusion (DSF) (ablation) | FSRU · Weibo | Accuracy: 0.876 -> 0.901 | Helped |
| add dual contrastive learning (CL) (ablation) | FSRU · Weibo | Accuracy: 0.889 -> 0.901 | Helped |
| add unimodal spectrum compression (USC) (ablation) | FSRU · Twitter | Accuracy: 0.910 -> 0.952 | Helped |
| add cross-modal spectrum co-selection (CSC) (ablation) | FSRU · Twitter | Accuracy: 0.924 -> 0.952 | Helped |
| add distribution similarity-based fusion (DSF) (ablation) | FSRU · Twitter | Accuracy: 0.947 -> 0.952 | Helped |
| add dual contrastive learning (CL) (ablation) | FSRU · Twitter | Accuracy: 0.937 -> 0.952 | Helped |
Surprising findings
- Removing the distribution-similarity fusion or the contrastive learning causes only slight drops, suggesting the spectrum modules carry most of the gain.
- The spatial-MLP variant converges worse than self-attention but is cheaper, a trade-off the authors note.
Experimental setup
Datasets: Twitter Weibo
Benchmarks: Twitter Weibo
Metrics: Accuracy Precision Recall F1
Baselines: att-RNN EANN MVAE SpotFake HMCAN CAFE BMR LogicDM
Builds on: GFNet InfoNCE CAFE BMR
Closest papers by meaning
- External Reliable Information-enhanced Multimodal Contrastive Learning for Fake News Detection (AAAI 2025 · similarity 0.83)
- Learning Complex Heterogeneous Multimodal Fake News via Social Latent Network Inference (AAAI 2025 · similarity 0.83)
- DAMMFND: Domain-Aware Multimodal Multi-view Fake News Detection (AAAI 2025 · similarity 0.83)
- Reinforced Adaptive Knowledge Learning for Multimodal Fake News Detection (AAAI 2024 · similarity 0.82)
- RaCMC: Residual-Aware Compensation Network with Multi-Granularity Constraints for Fake News Detection (AAAI 2025 · similarity 0.82)
- Structured Spectral Reasoning for Frequency-Adaptive Multimodal Recommendation (NeurIPS 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.