Research map / Vision-language and multimodal / Cross-modal learning
Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ITHP, a hierarchical variational information-bottleneck model that uses a prime modality as input and other modalities as detectors, achieving state-of-the-art sarcasm detection and sentiment results on MUStARD, CMU-MOSI and CMU-MOSEI.
Problem
Multimodal fusion methods combine all modalities identically and ignore the hierarchical, primary-to-secondary structure of information processing; the paper targets compact, non-redundant multimodal representations for sarcasm detection and sentiment analysis.
Approach
Designate a prime modality as the sole input and chain latent bottleneck states through the remaining modalities, training each level with a variational information bottleneck loss (minimize I(X0;B0) and I(B0;B1), maximize I(B0;X1) and I(B1;X2)) on top of standard task loss.
Key results
- On CMU-MOSI with DeBERTa, ITHP reports 88.7% binary accuracy, 88.6% F1, 0.643 MAE and 0.852 Pearson correlation, exceeding the reported human-level numbers (85.7, 87.5, 0.710, 0.820).
- On MUStARD three-modality (V-T-A) sarcasm detection, ITHP reaches 75.3 precision and 75.2 F-score versus 71.9 and 71.5 for MSDM.
- On CMU-MOSEI, ITHP achieves 87.3% binary accuracy and 0.813 correlation, outperforming MMIM and MAG baselines with DeBERTa or BERT encoders.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ITHP instead of MSDM for three-modality V-T-A sarcasm detection | MUStARD · MUStARD V-T-A | F-Score: 71.5 -> 75.2 | Helped |
| Use ITHP instead of the strongest listed BERT/DeBERTa baseline | CMU-MOSI · CMU-MOSI | BA: 86.1 -> 88.7 | Helped |
| Use ITHP instead of the strongest listed BERT/DeBERTa baseline | CMU-MOSI · CMU-MOSI | MAE: 0.649 -> 0.643 | Helped |
| Use ITHP instead of the strongest listed baseline | CMU-MOSEI · CMU-MOSEI | BA: 86.0 -> 87.3 | Helped |
| Use ITHP instead of the strongest listed baseline | CMU-MOSEI · CMU-MOSEI | MAE: 0.526 -> 0.564 | Hurt |
| Set the ITHP Lagrange multipliers to beta = 32 and gamma = 8 rather than beta = 2 and gamma = 2 (ablation) | ITHP sarcasm detection · MUStARD V-T-A | weighted precision: 0.619 -> 0.753 | Helped |
| Set the ITHP Lagrange multipliers to beta = 32 and gamma = 8 rather than beta = 2 and gamma = 2 (ablation) | ITHP sarcasm detection · MUStARD V-T-A | recall: 0.607 -> 0.752 | Helped |
Surprising findings
- Self-MM with DeBERTa features degrades sharply (e.g., 55.1% BA on CMU-MOSI), suggesting dependence on BERT-specific feature extraction.
- Reported hyperparameter sweeps show the worst case (beta=2, gamma=2) performs even worse than unimodal learning on sarcasm detection.
Limitations
- Relies on a preset ordering of modalities chosen by prior knowledge; exhaustive ordering search is time-consuming.
- Performance may suffer when the prime modality lacks sufficient information on its own and needs complementary modalities.
- Sarcasm experiments use fixed pre-extracted embeddings and a small two-layer MLP predictor.
Experimental setup
Datasets: MUStARD CMU-MOSI CMU-MOSEI
Benchmarks: MUStARD CMU-MOSI CMU-MOSEI
Metrics: weighted precision weighted recall F-score binary accuracy F1 MAE Pearson correlation
Baselines: MSDM Self-MM MMIM MAG MIB BBFN UniMSE
Builds on: information bottleneck deep variational information bottleneck DeBERTa MAG-BERT
Closest papers by meaning
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.86)
- SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning (NeurIPS 2026 · similarity 0.86)
- CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learning (NeurIPS 2025 · similarity 0.86)
- Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems (NeurIPS 2025 · similarity 0.86)
- Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.85)
- Balancing Multimodal Training Through Game-Theoretic Regularization (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.