Research map / Vision-language and multimodal / Cross-modal learning
ReconBoost: Boosting Can Achieve Modality Reconcilement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReconBoost, a modality-alternating boosting paradigm that updates one modality learner at a time with a KL-based reconcilement regularizer, shown to resemble gradient boosting, to mitigate modality competition in multi-modal learning.
Problem
Modality competition in joint multi-modal learning, where a dominant modality overpowers the gradient updates and prevents effective exploitation of weak-modality features.
Approach
Replace synchronous joint optimization with modality-alternating updates, where each round updates one modality learner with a KL-based reconcilement regularizer against the other modalities' predictions; this is shown equivalent to Friedman's gradient boosting with only the newest learner per modality kept, plus memory consolidation and global rectification schemes.
Key results
- Ours achieves 71.35% on AVE and 79.82% on CREMA-D accuracy, outperforming UMT (67.71% and 70.97%).
- On ModelNet40, Ours reaches 91.78% vs UMT 90.07%.
- On CH-SIMS, Ours reaches 73.88% vs best baseline UME 71.77%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReconBoost instead of UMT | multimodal learning with ResNet-18 · AVE | accuracy: 67.71 -> 71.35 | Helped |
| Use ReconBoost instead of UMT | multimodal learning with ResNet-18 · CREMA-D | accuracy: 70.97 -> 79.82 | Helped |
| Use ReconBoost instead of UMT | multimodal learning with ResNet-18 · ModelNet40 | accuracy: 90.07 -> 91.78 | Helped |
| Use ReconBoost instead of UMT | multimodal learning · MOSEI | accuracy: 67.04 -> 68.61 | Helped |
| Use ReconBoost instead of UME | multimodal learning · MOSI | accuracy: 76.97 -> 77.96 | Helped |
| Use ReconBoost instead of UME | multimodal learning · CH-SIMS | accuracy: 71.77 -> 73.88 | Helped |
| Use ReconBoost instead of HSR | retrieval on multimodal features · AVE | MAP: 41.49 -> 43.85 | Helped |
| Use ReconBoost instead of HSR | retrieval on multimodal features · CREMA-D | MAP: 55.22 -> 60.52 | Helped |
| Use a moderate reconcilement regularization coefficient; too much regularization can lower it (ablation) | ReconBoost · CREMA-D and MOSEI | accuracy | Mixed |
| Tune the memory consolidation coefficient α (ablation) | ReconBoost · CREMA-D and MOSEI | accuracy | Helped |
| Add global rectification scheme (ablation) | ReconBoost · CREMA-D | accuracy | Helped |
| Add global rectification scheme (ablation) | ReconBoost · AVE | accuracy | Helped |
Surprising findings
- Modality-specific encoders trained by ReconBoost surpass uni-modal training on the weak modality (e.g., visual 73.01% vs 50.14% on CREMA-D).
Limitations
- Relies on modality-alternating schedule and hyperparameters (lambda, alpha, T1, T2) that require tuning per dataset.
- Evaluated mainly on small-scale benchmarks with pre-extracted or ResNet-18 features rather than large foundation models.
Experimental setup
Datasets: AVE CREMA-D ModelNet40 MOSEI MOSI CH-SIMS
Benchmarks: AVE CREMA-D ModelNet40 MOSEI MOSI CH-SIMS
Metrics: Accuracy (%) mAP (%) modality imbalance ratio
Baselines: Concat Fusion G-Blending OGM-GE PMR UME UMT CMCL HSR
Builds on: Gradient Boosting On-the-fly Gradient Modulation (OGM-GE) Prototypical Modal Rebalance (PMR) Uni-modal Teacher (UMT) Gradient Blending (G-Blending)
Closest papers by meaning
- Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning (NeurIPS 2025 · similarity 0.88)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.88)
- Improving Multimodal Learning Balance and Sufficiency through Data Remixing (ICML 2025 · similarity 0.88)
- A Closer Look at Multimodal Representation Collapse (ICML 2025 · similarity 0.87)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.87)
- Learning Modality Knowledge Alignment for Cross-Modality Transfer (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.