AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework

Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard J. Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Ruslan Salakhutdinov, Louis-Philippe Morency

NeurIPS 2023 · Vision-language and multimodal › Human and embodied modeling · approach: Multimodal and affective interaction modeling

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Compute PID in polar coordinates instead of CartesianCVX PID estimator · Gaussian mixture modelPID redundancy: redundancy is 0 under PolarMixed
Use CVX versus Batch estimatorPID estimators · Synthetic D_Rredundancy R: 0.16 vs 0.29 (truth 0.58)No clear effect
Increase ratio of z1 in label generationPID estimates · Synthetic generativeU1: 0.01 -> 0.06Improved
Use synergistic higher-order fusion (Tensor) rather than additive fusionMultimodal models · D_S accuracyaccuracy: 0.66 -> 0.74Improved
Use early fusion (EF) on synergy datasetEF · D_S accuracyaccuracy: 0.56Hurt
Use uniqueness-oriented methods (Lower, Rec) vs additive/agreeMultimodal models · D_Uuniqueness U: 0.19 -> 0.55Improved
Use PID agreement between dataset and model as proxyPID agreement · 10 synthetic datasetscorrelation with accuracy: rho = 0.81Improved
Missing modality: relate drop to U_iMultimodal models · Robustnesscorrelation of performance drop: rho=0.8Improved
For low U_i, relate S to drop vs RPID · Robustness (U_i<=0.05)correlation of performance drop: 0.73 vs 0.01Mixed
Select models via nearest synthetic dataset by PIDModel selection · Real-world datasets% of best performance: 95.15% -> 100%Improved
Compare PID with human annotationCVX/Batch · MultiBench datasetsagreement of dominant interaction: 4 datasets agreeImproved
Add unimodal-genomic info on TCGA-LGGPID · TCGA-LGGU2: 0.06No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy