Research map / Vision-language and multimodal / Human and embodied modeling
Efficient Multimodal Dataset Distillation via Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EDGE, a generative (fine-tuned Stable Diffusion) multimodal dataset distillation method with bi-directional contrastive and diversity losses plus MLLM caption synthesis, distilling image-text datasets far faster than MTT-based methods.
Problem
Multimodal image-text dataset distillation with matching training trajectories is extremely slow and memory-hungry, and directly using generative models yields poor image-text correspondence and low diversity.
Approach
Fine-tune a latent diffusion model with an InfoNCE-style bi-directional contrastive loss between denoised image latents and text embeddings, plus a diversity loss on concatenated image-text features; sample images conditioned on real captions and add MLLM-generated extra captions per image.
Key results
- On COCO with 500 pairs, EDGE reaches IR@1 1.8 / TR@1 2.9, competitive with LoRS (IR@1 2.1 / TR@1 2.8), while LoRS is far more expensive.
- On COCO with 1000 pairs, EDGE achieves IR@1 2.8 and TR@1 3.9, outperforming LoRS (2.5 / 3.6) and using 7.7-8.5 GPU hours vs 151-156 for LoRS at 500-1000 pairs (about 18x faster).
- On Flickr30K with 500 pairs, LoRS outperforms EDGE (IR@1 10.0 vs 6.7), so EDGE is not uniformly best; EDGE is reported as the only applicable distillation method on CC3M with 1000 pairs and a single GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EDGE instead of LoRS for 500-pair distillation | Flickr30K · Flickr30K, 500 pairs | IR@1: 10.0 -> 6.7 | Hurt |
| Use EDGE instead of LoRS for 500-pair distillation | COCO · COCO, 500 pairs | TR@1: 2.8 -> 2.9 | Helped |
| Use EDGE instead of random selection | CC3M · CC3M, 1000 pairs | IR@1: 0.1 -> 0.2 | Helped |
| Use EDGE instead of LoRS for 1000-pair distillation | COCO · COCO, 1000 pairs | IR@1: 2.5 -> 2.8 | Helped |
| Use EDGE instead of LoRS for 1000-pair distillation | Flickr30K · Flickr30K, 1000 pairs | IR@1: 11.2 -> 9.9 | Hurt |
| Use EDGE instead of LoRS | Flickr30K distillation · Flickr30K, 500 pairs | GPU hours: 54.2 -> 13.6 | Helped |
| Add contrastive loss to the Stable Diffusion v1.5 baseline (ablation) | COCO distillation · COCO, 500 pairs | IR@1: 1.2 -> 1.3 | Helped |
| Add diversity loss to the contrastive-loss workflow (ablation) | COCO distillation · COCO, 500 pairs | IR@1: 1.3 -> 1.4 | Helped |
| Add caption synthesis to EDGE losses (ablation) | COCO distillation · COCO, 500 pairs | IR@1: 1.4 -> 1.8 | Helped |
| Replace Stable Diffusion v1.5 with Stable Diffusion 3 (ablation) | COCO distillation · COCO, 500 pairs | IR@1: 1.2 -> 1.2 | No clear effect |
| Increase caption count from 1 to 2 per image (ablation) | COCO distillation · COCO, 500 image-text pairs | IR@1: 1.9 -> 2.8 | Helped |
| Increase caption count from 2 to 5 per image (ablation) | COCO distillation · COCO, 500 image-text pairs | IR@1: 2.8 -> 1.2 | Hurt |
Surprising findings
- Plain Stable Diffusion v1.5 samples yield low image-text retrieval performance, showing vanilla generative models do not preserve image-text correspondence.
- Increasing captions per image (CPI=5) with a fixed pair budget hurts retrieval, since fewer unique images are kept.
Limitations
- Does not outperform LoRS on Flickr30K at 500 pairs and on COCO IR at 500 pairs
- CC3M evaluation uses only 2.3M of 3.3M pairs because of unavailable URLs, and a 1000-pair validation subset
- Caption synthesis depends on MLLM choice; increasing captions per image reduces unique images and hurts performance
Experimental setup
Datasets: Flickr30K COCO CC3M
Benchmarks: Flickr30K COCO CC3M
Metrics: IR@1 IR@5 IR@10 TR@1 TR@5 TR@10 CLIP score FID GPU hours
Baselines: MTT-VL LoRS TESLA Stable Diffusion v1.5 Rand HERD K-Center Forget
Builds on: Matching Training Trajectories InfoNCE Stable Diffusion LoRS Minimax Diffusion
Closest papers by meaning
- CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder (NeurIPS 2025 · similarity 0.88)
- Low-Rank Similarity Mining for Multimodal Dataset Distillation (ICML 2024 · similarity 0.88)
- TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models (ICLR 2025 · similarity 0.87)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.87)
- Active Data Curation Effectively Distills Large-Scale Multimodal Models (CVPR 2025 · similarity 0.87)
- MGD$^3$ : Mode-Guided Dataset Distillation using Diffusion Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.