Research map / Learning theory and algorithms / Prediction and decision losses
Group Distributionally Robust Dataset Distillation with Risk Minimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Robust Dataset Distillation (RDD), which adds a clustering-based group DRO loss (CVaR average and max over clusters of real data assigned to synthetic points) to the network-update step of matching-based dataset distillation methods such as IDC and GLaD.
Problem
Dataset distillation produces synthetic data that may generalize poorly to low-density subgroups and domain shifts; existing matching-based methods optimize average empirical loss and ignore subgroup coverage and robustness.
Approach
Cluster real training samples by their nearest synthetic point and, during network updates, optimize a CVaR-based group DRO objective (average plus max CVaR across clusters) as a plug-in to existing matching-based DD methods, with the synthetic set then updated by the original matching loss.
Key results
- On CIFAR-10 with IPC 10, GLaD improves from 46.7% to 50.2% with the robust CVaR loss, and IDC from 67.5% to 68.6%.
- On ImageNet-10 with IPC 10, GLaD improves from 50.9% to 55.2%; cluster-min accuracy under Invert shift rises from 17.0% to 21.6%.
- On MetaShift the worst-group accuracy rises from 51.3% to 57.0% when GLaD is combined with the robust loss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RDD, including clustered group CVaR optimization, to GLaD | GLaD · CIFAR-10, IPC 10, standard test | top-1 accuracy: 46.7 -> 50.2 | Helped |
| Apply RDD, including clustered group CVaR optimization, to IDC | IDC · ImageNet-10, IPC 10, standard test | top-1 accuracy: 72.8 -> 74.6 | Helped |
| Apply RDD to GLaD | GLaD · ImageNet-10, Cluster-min | top-1 accuracy: 34.9 -> 47.1 | Helped |
| Apply RDD to GLaD | GLaD · MetaShift | worst-group accuracy: 51.3 -> 57.0 | Helped |
| Apply RDD to GLaD | GLaD · ImageNetBG | average accuracy: 41.7 -> 45.5 | Helped |
| Add average CVaR loss alongside cross entropy (ablation) | GLaD · CIFAR-10, IPC 10 | accuracy: 46.7 -> 49.1 | Helped |
| Add maximum CVaR loss alongside cross entropy (ablation) | GLaD · CIFAR-10, IPC 10 | accuracy: 46.7 -> 47.9 | Helped |
| Use both average and maximum CVaR losses with cross entropy (ablation) | GLaD · CIFAR-10, IPC 10 | Cluster-min accuracy: 40.2 -> 46.7 | Helped |
| Vary CVaR ratio α; performance peaks at α=0.8, while α=1.0 and small α reduce performance (ablation) | GLaD-RDD · standard test and Cluster-min | accuracy | Mixed |
| Increase cluster count above 10 (ablation) | GLaD-RDD · CIFAR-10, IPC 50, standard accuracy | accuracy: 62.5 -> 60.9; 62.5 -> 59.9 | Hurt |
| Vary mini-batch size from 256 to 64 or 128 (ablation) | GLaD-RDD · CIFAR-10, IPC 10 | Cluster-min accuracy: 46.7 -> 40.3; 46.7 -> 45.1 | Hurt |
| Initialize synthetic samples with clustering centers instead of random real samples (ablation) | GLaD-RDD · CIFAR-10, IPC 10 | accuracy: 50.2 -> 50.3 | No clear effect |
Surprising findings
- The theoretical guarantees rely on loosely stated assumptions and proofs that are informal (e.g., LDP and Kolmogorov 0-1 arguments), so the theory is weakly connected to the empirical method.
Limitations
- Theoretical analysis is informal and relies on strong assumptions (finite/compact support, asymptotic regimes) and is only loosely linked to the algorithm.
- Gains depend on the number of clusters and CVaR ratio alpha; performance drops when clusters are too many relative to IPC.
- Evaluated on image classification benchmarks only, with the robustness tests relying on synthetic corruptions and clustering of test sets.
Experimental setup
Datasets: CIFAR-10 SVHN ImageNet-10 ImageNet-A Tiny-ImageNet MetaShift ImageNetBG CUB-200
Benchmarks: CIFAR-10 SVHN ImageNet-10 Tiny-ImageNet MetaShift ImageNetBG CUB-200 ImageNet-A ImageNet-B
Metrics: top-1 accuracy cluster-min accuracy worst-group accuracy
Baselines: Random DSA DM KIP IDC GLaD DREAM PDD IDM MMD RSC HYPO
Builds on: GLaD IDC DSA DM KIP
Closest papers by meaning
- Distilling Long-tailed Datasets (CVPR 2025 · similarity 0.88)
- Towards Adversarially Robust Dataset Distillation by Curvature Regularization (AAAI 2025 · similarity 0.88)
- Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.88)
- Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching (ICLR 2024 · similarity 0.87)
- Embarrassingly Simple Dataset Distillation (ICLR 2024 · similarity 0.87)
- DD-Ranking: Rethinking the Evaluation of Dataset Distillation (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.