Research map / Learning theory and algorithms / Prediction and decision losses

Efficient Core-set Selection for Deep Learning Through Squared Loss Minimization

ICML 2025 · Learning theory and algorithms › Prediction and decision losses · approach: Consistency of surrogate losses

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MRMC, a core-set selection criterion derived from minimizing the sum of squared losses, which scores samples by their loss reduction under brief early training and adds a proxy-based diversity regularizer.

Problem

Core-set selection for deep learning needs to balance efficiency and effectiveness, but heuristic methods lack rigor and optimization-based methods (bi-level) are computationally expensive.

Approach

Replace the bi-level core-set objective with minimizing the sum of squared losses over all samples, which decomposes into maximizing total loss reduction (MRMC: select samples with largest loss reduction after R epochs of training, modeled with an exponential fit) and a balance regularizer implemented via a linear proxy model that favors samples poorly fit by the core-set.

Key results

• MRMC-R reaches 94.61% average accuracy on CIFAR-10 under the full-training protocol versus 93.04% for Random and 93.05% for Glister (averaged over 70/50/30% ratios).
• On CIFAR-100 under the early training protocol, MRMC-R averages 73.43% versus 71.94% for Moderate and 59.25% for EL2N.
• On ImageNet-1K at 50% selection, MRMC-R reaches 69.85% versus 68.51% for Random.

Measured effects reported in the paper

Surprising findings

• Under the early training protocol, EL2N and Dyn-Unc degrade sharply on CIFAR-100 and Tiny-ImageNet at small core-set sizes, while diversity-aware methods hold up.
• Moderate and BoundaryCCS outperform on CIFAR-10 under early training, suggesting full training may overfit on simpler tasks.

Limitations

• Relies on assumptions of stability and symmetry of loss attribution values, validated only on a small 50-sample case study.
• Regularization introduces multiple hyperparameters (R, rho, gamma) whose optimal values vary with core-set size and task.
• Some competitor methods were reproduced without the label balance constraint, and MRMC-R underperforms on ImageNet-1K at 30% and 10% ratios.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1k

Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1k

Metrics: test accuracy selection time

Baselines: Random EL2N GraNd Glister CCS Moderate Dyn-Unc BoundaryCCS D2pruning

Builds on: EL2N Data Shapley CRAIG Moderate-DS D2Pruning

Closest papers by meaning

• Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraints (ICML 2024 · similarity 0.88)
• Balancing Feature Similarity and Label Variability for Optimal Size-Aware One-shot Subset Selection (ICML 2024 · similarity 0.87)
• Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes (NeurIPS 2025 · similarity 0.86)
• OPTICAL: Leveraging Optimal Transport for Contribution Allocation in Dataset Distillation (CVPR 2025 · similarity 0.84)
• Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets? (NeurIPS 2026 · similarity 0.84)
• Multinoulli Extension: A Lossless Yet Effective Probabilistic Framework for Subset Selection over Partition Constraints (ICML 2025 · similarity 0.84)