Research map / Optimization / Optimizers
Revisit Micro-batch Clipping: Adaptive Data Pruning via Gradient Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reframes micro-batch clipping as adaptive data pruning of "dragger" samples, proves an asymptotically faster convergence rate with a micro-batch-size-dependent bias, and reports gains on ASR, ViT and T5 with a weaker effect on unbalanced multi-domain data.
Problem
Micro-batch clipping has been reported to improve ASR models only at specific micro-batch sizes, and its underlying mechanism is unexplained.
Approach
Model harmful "dragger" per-example gradients as orthogonal to benign gradients; under this assumption, micro-batch clipping (adaptive clipping to the minimal micro-batch gradient norm) asymptotically accelerates convergence but adds a constant bias that depends on micro-batch size, explaining sweet-spot micro-batch sizes.
Key results
- On LibriSpeech with micro-batch size 4, WER improves from 1.93 to 1.85 on test-clean and from 3.99 to 3.80 on test-other, while b=1 and b=512 degrade performance.
- On ImageNet, DeiT-B with micro-batch clipping reaches 83.3% Top-1 and 96.8% Top-5 versus 81.8% and 95.8% for the baseline.
- On SuperGLUE with T5, weighted average accuracy rises only from 85.9 to 86.0, with ReCoRD improving 83.9 to 84.3 while smaller subsets such as BoolQ, CB and RTE drop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive micro-batch clipping with micro-batch size 4 instead of the baseline | 600M Conformer XL · LibriSpeech test-clean | WER: 1.93 $\pm$ 0.04 -> \textbf{1.85} $\pm$ 0.07; 4.1\% relatively | Helped |
| Use adaptive micro-batch clipping with micro-batch size 4 instead of the baseline | 600M Conformer XL · LibriSpeech test-other | WER: 3.99 $\pm$ 0.01 -> \textbf{3.80} $\pm$ 0.05; 4.8\% relatively | Helped |
| Use micro-batch size 1 instead of 4 (ablation) | Micro-batch clipping on 600M Conformer XL · LibriSpeech test-clean | WER: 1.93 $\pm$ 0.04 -> 2.40 $\pm$ 0.13 | Hurt |
| Use micro-batch size 1 instead of 4 (ablation) | Micro-batch clipping on 600M Conformer XL · LibriSpeech test-other | WER: 3.99 $\pm$ 0.01 -> 4.86 $\pm$ 0.24 | Hurt |
| Use micro-batch size 512 instead of 4 (ablation) | Micro-batch clipping on 600M Conformer XL · LibriSpeech test-clean | WER: 1.93 $\pm$ 0.04 -> 1.94 $\pm$ 0.00 | Hurt |
| Use micro-batch size 512 instead of 4 (ablation) | Micro-batch clipping on 600M Conformer XL · LibriSpeech test-other | WER: 3.99 $\pm$ 0.01 -> 4.03 $\pm$ 0.03 | Hurt |
| Add adaptive micro-batch clipping | DeiT-B · ImageNet | Top-1 accuracy (%): 81.8 -> 83.3; 1.5\% | Helped |
| Add adaptive micro-batch clipping | DeiT-B · ImageNet | Top-5 accuracy (%): 95.8 -> 96.8; 1.0\% | Helped |
| Add adaptive micro-batch clipping | T5 · SuperGLUE validation, weighted average | Accuracy (%): 85.9 -> 86.0; 0.1\% | Helped |
| Add adaptive micro-batch clipping | T5 · SuperGLUE validation, ReCoRD | Accuracy (%): 83.9 -> 84.3; 0.4\% | Helped |
| Add adaptive micro-batch clipping | T5 · SuperGLUE validation, BoolQ | Accuracy (%): 93.4 -> 91.8 | Hurt |
| Add adaptive micro-batch clipping | T5 · ReCoRD only | Accuracy (%): 84.5 -> 84.8; 0.3\% | Helped |
Surprising findings
- Micro-batch clipping degrades ASR performance at micro-batch size 1 and 512, so gains occur only at a specific sweet-spot micro-batch size.
- On multi-domain unbalanced SuperGLUE data, micro-batch clipping hurts smaller subsets while helping the largest one.
Limitations
- Micro-batch clipping can hurt performance on multi-domain unbalanced training data, raising fairness concerns.
- Memory overhead of micro-batch clipping remains a challenge under model sharding.
- The constant bias term is not tight; the measured proxy c-hat does not fully explain the micro-batch size effect.
Experimental setup
Datasets: LibriSpeech LibriLight handcrafted canaries ImageNet SuperGLUE C4
Benchmarks: LibriSpeech test-clean LibriSpeech test-other ImageNet SuperGLUE
Metrics: word error rate Top-1 accuracy Top-5 accuracy accuracy
Baselines: standard SGD / non-clipped training DeiT-B baseline T5 baseline
Builds on: DP-SGD micro-batch clipping BEST-RQ Sorscher et al. data pruning
Closest papers by meaning
- What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights (NeurIPS 2024 · similarity 0.86)
- Zipformer: A faster and better encoder for automatic speech recognition (ICLR 2024 · similarity 0.85)
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.85)
- MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric (CVPR 2024 · similarity 0.85)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.85)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.