Research map / Vision-language and multimodal / Human and embodied modeling
Debiased Multimodal Understanding for Human Language Sequences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SuCI, a plug-in causal intervention module that treats subjects as confounders and uses backdoor adjustment with subject prototypes to debias multimodal language understanding models on MOSI, MOSEI, and UR-FUNNY.
Problem
Multimodal language understanding models learn subject-specific spurious correlations from training subjects, causing prediction bias and poor generalization to new subjects.
Approach
Model subjects as unobserved confounders in a causal graph and approximate backdoor adjustment P(Y|do(X)) via a learned dictionary of subject prototypes, aggregated with attention and prior weights (NWGM feature-level approximation), plugged into existing MLU models.
Key results
- SuCI improves MMIM on UR_FUNNY to 70.92% Acc2 (+1.39) and DMD to 70.84% (+2.14) over vanilla.
- On MOSEI, DMD+SuCI reaches 54.6% Acc7, 85.8% Acc2 and 85.7% F1, gains of 1.1, 1.7 and 1.7 points.
- In cross-dataset evaluation (MOSI-train to MOSEI-test), MMIM+SuCI gains 2.5 Acc7 points and DMD+SuCI gains 2.7 Acc7 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SuCI to the baseline model | MOSI sentiment models · MOSI | Acc7, Acc2, and F1: MulT: +1.2, +0.8, +0.9; MISA: +1.4, +1.5, +1.2; Self-MM: +0.4, +0.4, +0.5; MMIM: +0.6, +0.8, +0.7; DMD: +1.2, +1.3, +1.3 | Helped |
| Add SuCI to the baseline model | MOSEI sentiment models · MOSEI | Acc7, Acc2, and F1: DMD: +1.1%, +1.7%, and +1.7% | Helped |
| Add SuCI to MMIM | MMIM · UR_FUNNY punchline detection | Acc2: +1.39% | Helped |
| Add SuCI to the baseline model | MOSI/MOSEI models evaluated cross-dataset · MOSI train → MOSEI test and MOSEI train → MOSI test | Acc7, Acc2, and F1: MOSI→MOSEI DMD Acc7: 48.9 -> 51.6; MOSEI→MOSI MMIM Acc7: 39.3 -> 41.1 | Helped |
| Replace dynamic fusion with average pooling (ablation) | SuCI · MOSI, MOSEI, and UR_FUNNY | Acc2: MISA/DMD: 83.3/84.6 -> 82.9/84.2; 83.5/85.8 -> 83.2/85.3; 68.96/70.84 -> 68.64/70.67 | Hurt |
| Add the subject discriminator (ablation) | SuCI · MOSI, MOSEI, and UR_FUNNY | Acc2: MISA/DMD: 82.2/83.8 -> 83.3/84.6; 82.7/85.0 -> 83.5/85.8; 68.57/69.69 -> 68.96/70.84 | Helped |
| Add the task discriminator (ablation) | SuCI · MOSI, MOSEI, and UR_FUNNY | Acc2: MISA/DMD: 82.6/84.1 -> 83.3/84.6; 82.9/85.3 -> 83.5/85.8; 68.62/70.44 -> 68.96/70.84 | Helped |
| Keep text, visual, and audio modalities in confounder learning (ablation) | SuCI · MOSI, MOSEI, and UR_FUNNY | Acc2: MISA/DMD with text removed: 82.4/83.6, 82.7/84.6, 68.25/69.38; visual removed: 82.9/84.3, 83.1/85.4, 68.62/69.94; audio | Helped |
| Use subject prototypes instead of random confounders (ablation) | SuCI · MOSI, MOSEI, and UR_FUNNY | Acc2: Random dictionary: 79.3/81.0, 78.2/79.9, 62.46/84.05; proposed: 83.3/84.6, 83.5/85.8, 68.96/70.84 | Helped |
| Use subject prototypes instead of clustered confounders (ablation) | SuCI · MOSI, MOSEI, and UR_FUNNY | Acc2: Clustered dictionary: 82.8/83.7, 82.9/85.0, 67.98/69.63; proposed: 83.3/84.6, 83.5/85.8, 68.96/70.84 | Helped |
| Add adaptive attention weights ψᵢ (ablation) | SuCI · MOSI, MOSEI, and UR_FUNNY | Acc2: Without ψᵢ: 82.7/84.2, 83.1/85.4, 68.45/70.32; proposed: 83.3/84.6, 83.5/85.8, 68.96/70.84 | Helped |
| Add prior probabilities p(zᵢ) (ablation) | SuCI · MOSI, MOSEI, and UR_FUNNY | Acc2: Without p(zᵢ): 83.0/84.5, 83.4/85.6, 68.73/70.66; proposed: 83.3/84.6, 83.5/85.8, 68.96/70.84 | Helped |
Surprising findings
- Random confounder dictionary can underperform vanilla methods, e.g., MOSI and MOSEI drop substantially with random Z.
- On UR_FUNNY, DMD with random Z reaches 84.05 Acc2, higher than the full SuCI result of 70.84, which is inconsistent with the other settings.
Limitations
- Subject labels are required to build confounder prototypes, limiting applicability when subject identity is unavailable.
- Gains on some settings are modest (often under 1 point), and results are reported as single numbers without variance.
Experimental setup
Datasets: MOSI MOSEI UR_FUNNY
Benchmarks: MOSI MOSEI UR_FUNNY
Metrics: Acc7 Acc2 F1 accuracy
Baselines: EF-LSTM LF-LSTM Graph-MFN TFN LMF MFM RAVEN MCTN TCSP PMR FDMER C-MFN
Builds on: MulT MISA Self-MM MMIM DMD
Closest papers by meaning
- Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark (NeurIPS 2025 · similarity 0.91)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.88)
- MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.88)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.88)
- Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.87)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.