Research map / Learning theory and algorithms / Optimal transport
OPTICAL: Leveraging Optimal Transport for Contribution Allocation in Dataset Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OPTICAL, a plug-and-play dataset distillation add-on that uses a Sinkhorn-approximated optimal transport matrix (on Gaussian-kernel cost) to allocate non-uniform contributions from real to synthetic samples, improving existing synthesis-based distillation methods.
Problem
Existing synthesis-based dataset distillation minimizes a homogeneous distance in which every real instance contributes equally to every synthetic sample, which ignores instance-level pairwise correspondence and intra-class variation.
Approach
Reformulate distance minimization as bi-level matching-and-approximating: estimate a real-to-synthetic contribution matrix via entropic optimal transport (Sinkhorn) using a cost derived from multi-scale Gaussian kernel relevance, then use this non-uniform allocation when optimizing the synthetic set with existing optimization-oriented or DM-based losses.
Key results
- Applied to M3D on CIFAR-10 (IPC 50 with ConvNet-3), OPTICAL improves accuracy from 69.9% to 71.0%, and with cross-architecture evaluation DANCE+Ours reaches 70.3% vs 64.8% on DenseNet-121 at IPC 50.
- On CIFAR-100 with IPC 10, DM+Ours gains are reported across methods; the largest gain shown is +4.3 points for IDM on CIFAR-100 at IPC 1 (20.1 to 24.4).
- Runtime overhead is modest: per-iteration time on CIFAR-10 IPC 50 rises from 0.066s to 0.097s for DM with OPTICAL on one RTX 4090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OPTICAL contribution allocation to DANCE | DANCE · MNIST, FashionMNIST, SVHN, CIFAR-10, and CIFAR-100; IPC 1, 10, and 50 | test accuracy: +0.3 to +1.5 | Helped |
| Add OPTICAL contribution allocation to DANCE | DANCE · TinyImageNet; IPC 1, 10, and 50 | test accuracy: +0.5; +0.5; +0.7 | Helped |
| Add OPTICAL contribution allocation to M3D | M3D · ImageNet-10; IPC 10 and 20 | test accuracy: +1.3; +1.7 | Helped |
| Use the proposed Gaussian kernel instead of N/A, a linear kernel, or a single Gaussian kernel (ablation) | DM+OPTICAL · CIFAR-10, IPC 10 | accuracy: 49.5; 47.8; 49.8 -> 50.2 | Helped |
| Vary the entropic regularization parameter 1/λ (ablation) | M3D+OPTICAL · CIFAR-100, IPC 50 | accuracy: 52.2; 52.8; 52.6; 53.0; 52.8; 53.0 | No clear effect |
| Vary Sinkhorn iterations T (ablation) | M3D+OPTICAL · CIFAR-100, IPC 50 | accuracy: 52.3; 52.5; 52.8; 53.0; 52.8; 53.0 | No clear effect |
Surprising findings
- Gains are larger in cross-architecture evaluation than in same-architecture evaluation, e.g., DANCE+Ours improves DenseNet-121 accuracy by +5.5 at IPC 50 when distilled with ConvNet-3.
- Gains are largest for DM, which uses the simplest distribution-matching loss, consistent with its instance-wise homogeneity.
Limitations
- Gains in many settings are small (often under 1 point), so improvements are incremental.
- Requires grid search for the entropic regularization parameter and kernel bandwidth choices.
- Adds extra per-iteration computation over the base distillation methods.
Experimental setup
Datasets: MNIST FashionMNIST SVHN CIFAR-10 CIFAR-100 TinyImageNet ImageNet-10 ImageNet-100
Benchmarks: MNIST FashionMNIST SVHN CIFAR-10 CIFAR-100 TinyImageNet ImageNet-10 ImageNet-100
Metrics: top-1 test accuracy
Baselines: DD LD DC KIP DSA MTT FTD ATT CAFE DataDAM DREAM DM
Builds on: DREAM DM IDM M3D DANCE
Closest papers by meaning
- D^4: Dataset Distillation via Disentangled Diffusion Model (CVPR 2024 · similarity 0.88)
- Large Scale Dataset Distillation with Domain Shift (ICML 2024 · similarity 0.88)
- Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality (ICLR 2024 · similarity 0.88)
- Taming Diffusion for Dataset Distillation with High Representativeness (ICML 2025 · similarity 0.87)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.87)
- Hyperbolic Dataset Distillation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.