Research map / Label-efficient and robust learning / Domain adaptation and generalization
MODfinity: Unsupervised Domain Adaptation with Multimodal Information Flow Intertwining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MODfinity, a multimodal unsupervised domain adaptation framework that treats labels as an independent modality to measure modal affinity and uses affinity-based teacher selection and distillation to control information flow at feature and label levels.
Problem
Multimodal unsupervised domain adaptation in which co-learning between modalities propagates low-quality pseudo-labels and features across modalities, causing error accumulation when sample quality varies across modalities and samples.
Approach
Model labels as an independent modality with a class encoder trained via a modal-affinity metric learning loss, use the affinity score to select per-sample teacher modality and pseudo-label, and distill features unidirectionally from teacher to student only on high-affinity samples.
Key results
- On AVE, MODfinity reaches 58.11% fusion accuracy versus 54.56% for the best prior baseline (MCT) and 44.13% source-only.
- On AVE with 80% of image samples replaced by Gaussian noise, MODfinity achieves 48.81% fusion accuracy vs. 43.02% source-only and 26.35% for CL.
- Ablation on AVE shows CE+MOD+MOML reaches 58.11% fusion versus 49.65% for CE alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply MODfinity instead of DLMM | Multimodal domain adaptation · AVE | accuracy (Fusion): 55.27 -> 58.11 | Helped |
| Apply MODfinity instead of DLMM | Multimodal domain adaptation · EPIC-Kitchens 55 (Mean) | accuracy (Fusion): 52.2 -> 55.6 | Helped |
| Apply MODfinity instead of MCT | Multimodal domain adaptation · CogBeacon (Mean) | accuracy (Fusion): 67.6 -> 69.9 | Helped |
| Add MOD to CE-only training (ablation) | MODfinity · AVE | accuracy (Image): 49.90 -> 53.22 | Helped |
| Add MOD to CE-only training (ablation) | MODfinity · AVE | accuracy (Fusion): 49.65 -> 54.46 | Helped |
| Add MOML to CE-only training (ablation) | MODfinity · AVE | accuracy (Image): 49.90 -> 51.17 | Helped |
| Add MOML to CE-only training (ablation) | MODfinity · AVE | accuracy (Fusion): 49.65 -> 54.55 | Helped |
| Add both MOD and MOML (ablation) | MODfinity · AVE | accuracy (Fusion): 49.65 -> 58.11 | Helped |
| Use CL instead of source-only training | AVE with 80% Gaussian-noise image samples · AVE noisy target domain | accuracy (Fusion): 43.02 -> 26.35 | Hurt |
| Apply MODfinity instead of source-only training | AVE with 80% Gaussian-noise image samples · AVE noisy target domain | accuracy (Fusion): 43.02 -> 48.81 | Helped |
Surprising findings
- Contrastive learning (CL) degrades performance on noisy AVE, dropping image accuracy and fusion well below source-only.
Limitations
- Framework currently requires separate tuning for each modality pair, limiting scalability to more modalities.
- Evaluated mainly on image and audio (and RGB/flow, EEG/facial keypoints) modalities.
Experimental setup
Datasets: AVE EPIC-Kitchens 55 CogBeacon
Benchmarks: AVE EPIC-Kitchens 55 CogBeacon
Metrics: accuracy
Baselines: Source-only CT DANN MM-SADA MO CL DLMM RNA C-RNA MCT
Builds on: MCT DLMM contrastive learning domain adaptation (CL) MM-SADA co-training (CT)
Closest papers by meaning
- Enhancing Multimodal Cooperation via Sample-level Modality Valuation (CVPR 2024 · similarity 0.87)
- Balancing Multimodal Training Through Game-Theoretic Regularization (NeurIPS 2025 · similarity 0.86)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.86)
- Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.86)
- Facilitating Multimodal Classification via Dynamically Learning Modality Gap (NeurIPS 2024 · similarity 0.86)
- SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.