Research map / Vision-language and multimodal / Cross-modal learning
Low-Rank Similarity Mining for Multimodal Dataset Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRS, which distills a learnable low-rank ground-truth image-text similarity matrix alongside synthetic image-text pairs (built on TESLA/MTT) to improve multimodal contrastive dataset distillation on Flickr30k and COCO.
Problem
Dataset distillation for image-text contrastive learning data is hard because such pairs lack categories and have high batch variance, and existing methods fix the image-text correspondence to identity.
Approach
Jointly learn a ground-truth similarity matrix for synthetic image-text pairs, parameterized as a diagonal plus a low-rank residual (omega*I + alpha/r * L R^T) to keep storage linear, and train contrastive models with continuous losses (weighted BCE or extended InfoNCE) that use this matrix.
Key results
- On Flickr30k with 100 synthetic pairs, LoRS wBCE reaches IR@1 8.3 and TR@10 49.2, versus 4.7 and 39.1 for MTT-VL at 100 pairs.
- At 200 pairs on Flickr30k, LoRS wBCE achieves IR@1 8.6 and TR@10 53.4 (MTT-VL: 4.6 and 41.9), with the paper reporting up to ~50% relative improvement.
- Storage overhead of the similarity matrix is reported as about 0.3% memory and 0.8% training time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add similarity mining with LoRS and weighted BCE instead of the MTT-VL baseline | Image-text dataset distillation · Flickr30k, 200 pairs | IR@1: 4.6$\pm$0.9 -> 8.6$\pm$0.3 | Helped |
| Add similarity mining with LoRS and weighted BCE instead of the MTT-VL baseline | Image-text dataset distillation · COCO, 200 pairs | IR@1: 1.7$\pm$0.1 -> 2.4$\pm$0.1 | Helped |
| Use weighted BCE rather than BCE for the similarity-mining loss (ablation) | Similarity mining · Flickr30k, 200 pairs | IR@1: 0.6 -> 7.4 | Helped |
| Use weighted BCE rather than eNCE for the similarity-mining loss (ablation) | Similarity mining · Flickr30k, 200 pairs | IR@1: 4.1 -> 7.4 | Helped |
| Use weighted BCE rather than BCE for the LoRS loss (ablation) | LoRS · Flickr30k, 200 pairs | IR@1: 0.1 -> 8.6 | Helped |
| Use weighted BCE rather than eNCE for the LoRS loss (ablation) | LoRS · Flickr30k, 200 pairs | IR@1: 8.2 -> 8.6 | Helped |
| Vary low-rank residual rank from 5 to 1 (ablation) | LoRS_wBCE · Flickr30k, 200 pairs | IR@1: 8.6 -> 7.2 | Hurt |
| Vary low-rank residual rank from 5 to 10 (ablation) | LoRS_wBCE · Flickr30k, 200 pairs | IR@1: 8.6 -> 7.8 | Hurt |
| Vary low-rank residual rank from 5 to 20 (ablation) | LoRS_wBCE · Flickr30k, 200 pairs | IR@1: 8.6 -> 7.9 | Hurt |
| Add low-rank components L and R (ablation) | LoRS_wBCE · Flickr30k, 200 pairs | IR@1: 7.4 -> 8.6 | Helped |
| Add low-rank components L and R and diagonal weights ω (ablation) | LoRS_wBCE · Flickr30k, 200 pairs | IR@1: 0.2 -> 8.6 | Helped |
| Learn synthetic images rather than fixing them during distillation (ablation) | LoRS_wBCE · Flickr30k, 200 pairs | IR@1: 4.9 -> 8.6 | Helped |
Surprising findings
- Learning only the similarity matrix with frozen image and text (no image or text learning) still performs above a random model on Flickr30k.
- Computing the similarity matrix from a pretrained CLIP instead of learning it yields poor retrieval, indicating distilled data need not be natural-looking.
Limitations
- Only the text embedding is distilled; direct distillation of text tokens is not yet feasible.
- The similarity matrix adds storage, and the rank choice trades storage against performance.
- The contrastive loss must be redesigned (wBCE or eNCE) to use the learned continuous similarity.
Experimental setup
Datasets: Flickr30k COCO
Benchmarks: Flickr30k COCO
Metrics: IR@1 IR@5 IR@10 TR@1 TR@5 TR@10
Baselines: Random coreset Herding K-center Forgetting MTT-VL TESLA_wBCE DC DM
Builds on: TESLA MTT CLIP InfoNCE
Closest papers by meaning
- Efficient Multimodal Dataset Distillation via Generative Models (NeurIPS 2025 · similarity 0.88)
- CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder (NeurIPS 2025 · similarity 0.87)
- Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models (CVPR 2025 · similarity 0.85)
- DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models (CVPR 2025 · similarity 0.85)
- CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.85)
- Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.