Research map / Architectures and efficiency / Pruning and efficient tuning
Exploring Learning Complexity for Efficient Downstream Dataset Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DLC, a training-free hardness score that estimates learning complexity by weight-masking a pre-trained model at many ratios, plus FlexRand, a randomized under-sampling strategy, for downstream dataset pruning in image classification and LLM instruction tuning.
Problem
Existing dataset pruning methods require training or fine-tuning on the full dataset to score samples, which is impractical for fine-tuning large pre-trained models.
Approach
Define learning complexity as the integral of classification loss over a learning path; approximate it training-free by masking pre-trained weights (smallest L1 norm first) at random ratios with Monte Carlo sampling, and under-sample with FlexRand (random picks from easy and hard intervals) instead of top-K to avoid distribution shift.
Key results
- On the image benchmark averaged over 5 datasets and 9 pruning ratios with ResNet-18, DLC+FlexRand reaches 60.45% vs 59.21% for random, while pruning time drops to 21.5s versus about 1033s for one-shot fine-tuning baselines (~35x faster total time).
- Ranking correlation between masking-based and optimization-based learning complexity averages Spearman rho of about 0.51.
- On MMLU with LLM instruction pruning at 50%, DLC beats random across Mistral 7B, Llama3 8B and Gemma2 9B for both instruction datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DLC + FlexRand instead of Random | image dataset pruning · 5 downstream image datasets, averaged over architectures and pruning ratios | accuracy (%): 59.21% to 60.45% | Helped |
| Use DLC scoring instead of training-based scoring | image dataset pruning · 5 downstream image datasets | pruning time: 35× | Helped |
| Use DLC + FlexRand instead of Random | instruction dataset pruning · MMLU with various base models and instruction datasets | accuracy | Helped |
| Use FlexRand instead of Random under-sampling (ablation) | DLC · 5 downstream image datasets | accuracy: +1.39% | Helped |
| Use top-K Easy or top-K Hard instead of random under-sampling (ablation) | dataset pruning · downstream image datasets | accuracy | Hurt |
| Randomly mask pre-training weights instead of masking by L1 norm (ablation) | DLC scoring · 5 downstream image datasets | ranking stability and accuracy | Hurt |
| Increase learning-path size up to 5 (ablation) | DLC scoring · 5 downstream image datasets | ranking correlation | Helped |
| Use small splitting values (γ < 0.5) (ablation) | FlexRand · data-poor image pruning regimes | accuracy | Helped |
| Transfer scoring from ResNet-18 to larger models (ablation) | DLC · 5 downstream image datasets | accuracy: +0.32% | Mixed |
| Use FlexRand instead of top-K sampling (ablation) | DLC · downstream image datasets | distribution shift | Helped |
Surprising findings
- Many existing score-based pruning methods (e.g., Herding, GraNd, EL2N) perform worse than random sampling in the downstream fine-tuning setting.
- Top-K under-sampling causes severe subset distribution shift (measured by MMD), which hurts accuracy even when the score is informative.
Limitations
- Performance depends on the quality of the pre-trained model; may fail when representations are not meaningful.
- Transferring scores from a small model (ResNet-18) to larger models loses some performance relative to native scoring.
- Hyperparameter gamma selection relies on a validation split (20%) and may not always be optimal.
Experimental setup
Datasets: CXRB10 DeepWeeds DTD FGVCAircraft Sketch Alpaca Cleaned Dolly & HH-RLHF ImageNet-1K
Benchmarks: CXRB10 DeepWeeds DTD FGVCAircraft Sketch MMLU
Metrics: classification accuracy pruning time (s) Spearman rank correlation MMD
Baselines: Random Herding kCG CD Forgetting Least Conf Entropy Margin GraNd EL2N SSP
Builds on: EL2N Forgetting DeepCore
Closest papers by meaning
- Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty (ICML 2025 · similarity 0.92)
- Effective pruning of web-scale datasets based on complexity of concept clusters (ICLR 2024 · similarity 0.89)
- Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.89)
- Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning (CVPR 2024 · similarity 0.88)
- Label-Efficient Dataset Pruning via Semi-Supervised Pseudo-Labeling (NeurIPS 2026 · similarity 0.88)
- InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.