Research map / Vision-language and multimodal / Cross-modal learning
Predictive Dynamic Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Predictive Dynamic Fusion (PDF), which predicts a calibrated Collaborative Belief (Mono- and Holo-Confidence from predicted true-class probability, plus relative uncertainty calibration) as decision-level fusion weights, motivated by a generalization-error bound.
Problem
Dynamic multimodal fusion methods lack theoretical guarantees and are unstable under changing modality quality in open environments.
Approach
Derive from a Rademacher-based generalization error bound that fusion weights should be negatively covariant with a modality's own loss and positively covariant with other modalities' losses; predict p_true (instead of loss) as Mono-Confidence and sum of other modalities' losses ratio as Holo-Confidence, form Co-Belief, and calibrate it with a relative distribution-uniformity ratio (asymmetric cal
Key results
- On MVSA with epsilon=10 noise, PDF reaches 63.09% average accuracy versus 61.28% for QMF (best competing method).
- On NYU Depth V2 without noise, PDF reaches 71.37% average accuracy versus 70.40% for TMC and 70.03% for late fusion.
- On CREMA-D with time-varying noise, PDF gets 61.40% average vs 61.20% for QMF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PDF instead of QMF | multimodal classification · MVSA, ε = 10.0 | average accuracy: 61.28 -> 63.09 | Helped |
| use PDF instead of QMF | multimodal classification · UMPC FOOD101, ε = 10.0 | average accuracy: 62.21 -> 62.83 | Helped |
| use PDF instead of Late fusion | multimodal classification · NYU Depth V2, ε = 10.0 | average accuracy: 60.55 -> 62.56 | Helped |
| use PDF instead of QMF | multimodal classification · CREMA-D, ε = 10.0 | average accuracy: 41.60 -> 47.84 | Helped |
| use PDF instead of QMF | CREMA-D with time-varying noise · CREMA-D time-varying noise | average accuracy: 61.20 -> 61.40 | Helped |
| add Holo-Confidence to Mono-Confidence (ablation) | PDF on MVSA · MVSA, ε = 10 | average accuracy: 60.84 -> 60.97 | Helped |
| add Mono-Confidence and Relative Calibration to Holo-Confidence (ablation) | PDF on MVSA · MVSA, ε = 10 | average accuracy: 61.22 -> 63.09 | Helped |
| add Mono-Confidence and Holo-Confidence to Relative Calibration (ablation) | PDF on MVSA · MVSA, ε = 10 | average accuracy: 60.11 -> 63.09 | Helped |
| add Relative Calibration to Mono-Confidence and Holo-Confidence (ablation) | PDF on MVSA · MVSA, ε = 10 | average accuracy: 60.97 -> 63.09 | Helped |
| add Mono-Confidence to Holo-Confidence and Relative Calibration (ablation) | PDF on MVSA · MVSA, ε = 10 | average accuracy: 62.11 -> 63.09 | Helped |
| add Holo-Confidence to Mono-Confidence and Relative Calibration (ablation) | PDF on MVSA · MVSA, ε = 10 | average accuracy: 62.13 -> 63.09 | Helped |
| use asymmetric calibration instead of Relative Calibration alone (ablation) | PDF on NYU Depth V2 · NYU Depth V2, ε = 10 | accuracy: 62.19 -> 62.56 | Helped |
Surprising findings
- Directly predicting the loss as fusion weight is less effective than predicting p_true, because loss values concentrate near zero for most samples.
Limitations
- Theoretical bound is derived for binary classification with a convex logistic loss, while experiments use multiclass settings.
- Gains over strongest baselines on clean data are small (e.g., 0.2 to 1 point on several datasets).
Experimental setup
Datasets: UPMC FOOD-101 MVSA NYU Depth V2 CREMA-D PIE
Benchmarks: UPMC FOOD-101 MVSA NYU Depth V2 CREMA-D PIE
Metrics: average accuracy worst accuracy GEB decreasing proportion
Baselines: Late fusion Concat TMC QMF DynMM
Builds on: QMF TMC DynMM
Closest papers by meaning
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.86)
- Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.85)
- Balancing Multimodal Training Through Game-Theoretic Regularization (NeurIPS 2025 · similarity 0.85)
- SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning (NeurIPS 2026 · similarity 0.85)
- Multimodal Representation Learning by Alternating Unimodal Adaptation (CVPR 2024 · similarity 0.85)
- FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.