Research map / Vision-language and multimodal / Cross-modal learning
Data-Efficient Multimodal Fusion on a Single GPU
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FuseMix, a mixup-style augmentation on frozen pre-computed latents of unimodal encoders, training lightweight adapters to align image-text and audio-text spaces on a single V100 GPU.
Problem
Multimodal alignment (shared latent space) usually requires massive paired data and many GPUs; the paper seeks a data- and compute-efficient way to fuse modalities.
Approach
Freeze pre-trained unimodal encoders, pre-compute their latents once, and train small MLP fusion adapters with a contrastive loss on FuseMix augmentations: latent mixup where the mixing coefficient is shared across modalities so augmented pairs stay positive.
Key results
- FuseMix(D,B) with ~5M pairs reaches 71.2 text-to-image R@1 on Flickr30K vs CLIP's 68.7 (400M pairs), using ~5 GPU days on a single V100.
- With CC3M only (3M pairs), FuseMix(D,B) gets 59.9 text-to-image R@1 vs CLIP's 54.3 under the same data.
- FuseMix improves over no augmentation (66.1 to 71.2 text-to-image R@1 on Flickr30K); larger unimodal encoders and larger batch sizes also help.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FuseMix instead of no augmentation (ablation) | image-text fusion with DINOv2 + BGE · Flickr30K test set | Recall@1: 66.1 -> 71.2 (text-to-image); 80.5 -> 84.8 (image-to-text) | Helped |
| replace Gaussian noise with FuseMix (ablation) | image-text fusion with DINOv2 + BGE · Flickr30K test set | Recall@1: 66.7 -> 71.2 (text-to-image); 81.5 -> 84.8 (image-to-text) | Helped |
| replace random quantization with FuseMix (ablation) | image-text fusion with DINOv2 + BGE · Flickr30K test set | Recall@1: 67.8 -> 71.2 (text-to-image); 82.6 -> 84.8 (image-to-text) | Helped |
| scale up the unimodal encoders (ablation) | FuseMix image-text fusion · Flickr30K test set | Recall@1 | Helped |
| increase batch size (ablation) | FuseMix image-text fusion · Flickr30K test set | Recall@1 | Helped |
| increase the quantity of training pairs (ablation) | FuseMix image-text fusion · Flickr30K test set | Recall@1 | Helped |
| use higher-quality human-annotated pairs instead of web pairs (ablation) | FuseMix image-text fusion · Flickr30K test set | Recall@1 | Helped |
| select more diverse training pairs (ablation) | FuseMix image-text fusion · Flickr30K test set | relative improvement in Recall@1: up to nearly 40% | Helped |
| train FuseMix(D,B) on 5M rather than 3M image-text pairs (ablation) | FuseMix image-text fusion · Flickr30K 1K test set, text-to-image | Recall@1: 59.9 -> 71.2 | Helped |
| train FuseMix(D,B) on 5M rather than 3M image-text pairs (ablation) | FuseMix image-text fusion · Flickr30K 1K test set, image-to-text | Recall@1: 74.4 -> 84.8 | Helped |
| train FuseMix(D,B) on 5M rather than 3M image-text pairs (ablation) | FuseMix image-text fusion · COCO 5K test set, text-to-image | Recall@1: 32.2 -> 46.3 | Helped |
Surprising findings
- Outperforms CLIP on Flickr30K text-to-image retrieval with ~80x fewer image-text pairs and ~600x fewer GPU days.
- Audio-to-image generation via aligning Whisper to CLIP's latent space conditions GLIDE on audio without any audio training for GLIDE.
Limitations
- Performance is bounded by the semantic information in the frozen unimodal encoders.
- Unimodal encoders are not fine-tuned during fusion.
- No suitable quantitative metric for audio-to-image generation; evaluated qualitatively.
Experimental setup
Datasets: COCO Visual Genome SBU Captions Conceptual Captions 3M AudioCaps Clotho Flickr30K
Benchmarks: Flickr30K COCO AudioCaps Clotho
Metrics: Recall@1 Recall@5 Recall@10
Baselines: CLIP ALIGN FILIP LiT 3T CoCa LAION-CLAP VALOR ONE-PEACE VAST ML-ACT MMT
Builds on: mixup CLIP ImageBind InfoNCE
Closest papers by meaning
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.89)
- UNIALIGN: Scaling Multimodal Alignment within One Unified Model (CVPR 2025 · similarity 0.86)
- ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining (NeurIPS 2026 · similarity 0.86)
- Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion (CVPR 2024 · similarity 0.86)
- AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment (CVPR 2025 · similarity 0.86)
- Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.