Research map / Architectures and efficiency / RNNs and normalization
Beyond neural scaling laws: beating power law scaling via data pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| keep hardest examples instead of easiest when initial dataset is large | Margin/EL2N data pruning · Perceptron theory and ResNet18 CIFAR-10 | test error | Mixed |
| prune more aggressively (smaller fraction kept) as initial dataset grows (Pareto optimal fraction) | Data pruning with a good metric · SVHN, CIFAR-10, ImageNet ResNets | test error vs pruned dataset size | Helped |
| use a less accurate pruning metric (probe-teacher angle 10 deg vs 20 deg) (ablation) | Margin pruning · Perceptron theory | minimum useful fraction kept: 24% -> 46% | Hurt |
| use a weak probe (4 epochs) instead of a strong probe (40 epochs) for EL2N (ablation) | EL2N pruning · SVHN | test error scaling | Hurt |
| prune pre-trained ViT's CIFAR-10 fine-tuning data to 10% | ViT pre-trained on ImageNet21K · CIFAR-10 fine-tuning | accuracy | No clear effect |
| pre-train ResNet50 on 50% pruned ImageNet | ResNet50 transfer to CIFAR-10 · CIFAR-10 fine-tuning | accuracy | No clear effect |
| self-supervised prototype metric instead of memorization for pruning | ImageNet ResNet pruning · ImageNet top-5 | top-5 accuracy | No clear effect |
| add class balancing (50% ratio) to metric pruning (ablation) | ImageNet pruning · ImageNet | accuracy | Helped |
| vary number of k-means clusters by an order of magnitude (ablation) | Self-supervised prototype metric · ImageNet | top-5 accuracy | No clear effect |
Closest papers by meaning
- How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective (NeurIPS 2024 · similarity 0.89)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.89)
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training (NeurIPS 2025 · similarity 0.89)
- Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty (ICML 2025 · similarity 0.89)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.89)
- GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.