Research map / Vision-language and multimodal / Cross-modal learning
Beyond Modality Collapse: Representation Blending for Multimodal Dataset Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RepBlend, a multimodal dataset distillation framework that counters "modality collapse" via representation blending (MixUp-style in embedding space) and symmetric projection trajectory matching, improving image-text retrieval on Flickr-30K and MS-COCO with faster distillation.
Problem
Multimodal dataset distillation (MDD) for image-text data produces distilled sets whose embeddings over-concentrate within each modality and drift apart across modalities ("modality collapse"), and existing trajectory-matching methods supervise the two modalities asymmetrically.
Approach
Attribute modality collapse to the interaction of over-compression in dataset distillation with contrastive cross-modal supervision; mitigate it by blending representations across synthetic instances (MixUp in representation space) to weaken dominant cross-modal supervision, and balance supervision by matching trajectories of both image and text projection heads rather than only the text head.
Key results
- On Flickr-30K with 100 pairs (NFNet+BERT), RepBlend reaches IR@1 11.5 and TR@10 55.5, versus LoRS at 8.3 and 49.2, i.e. +9.4 IR@10 gain at the 100-pair setting as reported in the abstract.
- Distillation speed improves from 11.5 s/iter (LoRS) to 1.71 s/iter, a 6.7x speedup, with peak GPU VRAM reduced from 21.78 GB to 10.17 GB.
- On MS-COCO with 100 pairs, IR@10 is 22.3 versus 12.2 for LoRS; on ImageNet-100 at ipc=1, Top-1 is 65.8% versus 2.5% for SRe2L.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RepBlend instead of LoRS | MDD with NFNet+BERT · Flickr-30K, 100 pairs | IR@10: 35.1 -> 44.5 | Helped |
| Apply RepBlend instead of LoRS | MDD with NFNet+BERT · Flickr-30K, 100 pairs | TR@10: 49.2 -> 55.5 | Helped |
| Apply RepBlend instead of LoRS | MDD with NFNet+BERT · MS-COCO, 100 pairs | IR@10: 12.2 -> 22.3 | Helped |
| Apply RepBlend instead of LoRS | MDD with NFNet+BERT · MS-COCO, 100 pairs | TR@10: 19.6 -> 28.0 | Helped |
| Apply RepBlend instead of LoRS | MDD with EfficientAT+BERT · AudioCaps, 500 pairs | AR@10: 36.7±0.2 -> 46.8±0.2 | Helped |
| Apply RepBlend instead of LoRS | MDD with EfficientAT+BERT · AudioCaps, 500 pairs | TR@10: 41.3±0.3 -> 54.1±0.4 | Helped |
| Replace semantic blending with Gaussian noise at λ = 0.01 (ablation) | RepBlend · Flickr-30K, 100 distilled pairs | IR@1: 1.9% | Hurt |
| Replace semantic blending with Gaussian noise at λ = 0.01 (ablation) | RepBlend · Flickr-30K, 100 distilled pairs | TR@1: 2.1% | Hurt |
| Remove Representation Blending (ablation) | RepBlend · Flickr-30K, 100, 200, and 500 pairs | retrieval performance (IR@1/5/10 and TR@1/5/10) | Hurt |
| Remove Symmetric Projection Trajectory Matching (ablation) | RepBlend · Flickr-30K, 100, 200, and 500 pairs | retrieval performance (IR@1/5/10 and TR@1/5/10) | Hurt |
| Use symmetric projection matching rather than asymmetric trajectory matching (ablation) | MDD distillation · Modality alignment | modality gap: 0.318 -> 0.290 | Helped |
| Use projection-head trajectories for both modalities instead of asymmetric image-encoder/text-projection trajectories | MDD distillation · Flickr-30K | distillation speedup: 6.7× | Helped |
Surprising findings
- Injecting random Gaussian noise into text representations reduces intra-modal similarity but increases the modality gap, so naive perturbation hurts performance beyond a small noise level.
Limitations
- Current MDD framework is limited to pair-level modeling, restricting fine-grained alignment between text tokens and visual objects.
- Insufficient cross-instance interaction limits representation expressiveness and further compression gains.
Experimental setup
Datasets: Flickr-30K MS-COCO AudioCaps ImageNet-100
Benchmarks: Flickr-30K MS-COCO AudioCaps ImageNet-100
Metrics: IR@K TR@K Top-1 accuracy Top-5 accuracy distillation speed (s/iter) peak GPU memory intra-modal similarity modality gap
Baselines: Random Herding K-Center Forgetting MTT-VL TESLA-VL LoRS SRe2L
Builds on: MTT-VL LoRS trajectory matching (MTT) MixUp
Closest papers by meaning
- CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder (NeurIPS 2025 · similarity 0.88)
- A Closer Look at Multimodal Representation Collapse (ICML 2025 · similarity 0.87)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.87)
- Cross-modal Representation Flattening for Multi-modal Domain Generalization (NeurIPS 2024 · similarity 0.87)
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.87)
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.