Research map / Vision-language and multimodal / Cross-modal learning
A Multi-Focus-Driven Multi-Branch Network for Robust Multimodal Sentiment Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MFMB-Net, a multi-branch network that pairs a two-stream (macro/micro) multi-focus fusion branch with a reconstruction branch to handle randomly missing modalities in multimodal sentiment analysis.
Problem
Multimodal sentiment analysis when one or more modalities (text, audio, visual) are randomly missing, where existing methods reconstruct features independently of fusion and fail to capture global and local feature relations.
Approach
A dual-stream fusion branch (macro-fusion with a coarse hub Transformer across modalities for global features; micro-fusion with a pseudo-Siamese BiLSTM/memory-gated network and text-guided coarse-to-fine MLP for local features) jointly trained with a reconstruction branch whose generative loss restores missing features.
Key results
- Under complete modality on CMU-MOSEI, reports Acc-7 54.2 and F1 85.1 (Acc-2 84.7/85.1) versus Self-MM's 53.6 Acc-7 and 82.8 F1 as the best baseline-level comparison.
- Under incomplete modality (AUILC), MFMB-Net reports MAE 1.115 and Corr 0.461 on CMU-MOSI, outperforming TFR-Net (1.156, 0.452).
- Ablation on CMU-MOSI shows removing the reconstruction module causes the largest drop, 1.4% in Acc-7 and 0.7% in Acc-5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MFMB-Net instead of the strongest listed baseline | CMU-MOSI complete · CMU-MOSI complete | Corr: 0.793 -> 0.798 | Helped |
| Use MFMB-Net instead of the strongest listed baseline | CMU-MOSEI complete · CMU-MOSEI complete | Acc-7: +0.6% in Acc-7 | Helped |
| Use MFMB-Net instead of the strongest listed baseline | CMU-MOSEI complete · CMU-MOSEI complete | F1: +2.2% in F1 | Helped |
| Use MFMB-Net instead of the strongest listed baseline | CMU-MOSI incomplete (random modality missing) · CMU-MOSI incomplete (AUILC) | Acc-2: 66.4/67.0 | Hurt |
| Use MFMB-Net instead of the strongest listed baseline | CMU-MOSI incomplete (random modality missing) · CMU-MOSI incomplete (AUILC) | MAE: 1.156 -> 1.115 | Helped |
| Use MFMB-Net instead of the strongest listed baseline | CMU-MOSEI incomplete (random modality missing) · CMU-MOSEI incomplete (AUILC) | Acc-5: +0.6% improvement in Acc-5 | Helped |
| Use MFMB-Net instead of the strongest listed baseline | CMU-MOSEI incomplete (random modality missing) · CMU-MOSEI incomplete (AUILC) | F1: +2.2% increase in the F1 score | Helped |
| Add macro fusion (ablation) | MFMB-Net · CMU-MOSI incomplete (AUILC) | Corr: 0.451 -> 0.461 | Helped |
| Add micro fusion (ablation) | MFMB-Net · CMU-MOSI incomplete (AUILC) | Corr: 0.461 -> 0.461 | No clear effect |
| Add reconstruction module (ablation) | MFMB-Net · CMU-MOSI incomplete (AUILC) | Corr: 0.422 -> 0.461 | Helped |
Surprising findings
- Acc-2 on CMU-MOSI is relatively weak compared with baselines, which the authors attribute to label imbalance and the low distinguishability of binary accuracy.
- Removing the macro-fusion stream has a smaller effect than removing micro-fusion, suggesting global coarse features carry less information density than local features.
Limitations
- Acc-2 on CMU-MOSI underperforms, attributed to class imbalance and binary metric limitations.
- Missing modalities are simulated by random masking rather than real-world sensor failures.
- Only two benchmarks (CMU-MOSI, CMU-MOSEI) evaluated.
Experimental setup
Datasets: CMU-MOSI CMU-MOSEI
Benchmarks: CMU-MOSI CMU-MOSEI
Metrics: MAE Corr Acc-7 Acc-5 Acc-2 F1 AUILC
Baselines: TFN LMF MulT MISA Self-MM TFR-Net MMIM
Builds on: TFR-Net MMIN MulT Self-MM MISA
Closest papers by meaning
- A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities (AAAI 2024 · similarity 0.90)
- Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning (NeurIPS 2024 · similarity 0.89)
- DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.88)
- Towards Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2024 · similarity 0.87)
- Towards Multimodal Sentiment Analysis via Hierarchical Correlation Modeling with Semantic Distribution Constraints (AAAI 2025 · similarity 0.87)
- SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.