Research map / Architectures and efficiency / RNNs and normalization
MERIT: Maximum-normalized Element-wise Ratio for Language Model Large-batch Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MERIT, a LAMB-style large-batch optimizer that uses max-norm trust ratios and element-wise row/column ratios with clipping to curb max attention logit growth, evaluated on GPT-2 and Llama pretraining on OpenWebText and C4.
Problem
Large-batch training of language models with AdamW and LAMB degrades generalization because max attention logits sharply increase, causing attention entropy collapse and worse validation loss.
Approach
Replace LAMB's l2-norm trust ratio with a max-norm trust ratio that directly bounds query/key weight maxima, and refine it into element-wise ratios taken as the max of row-wise, column-wise and weight-wise ratios, followed by element-wise update clipping to 1.
Key results
- On GPT-2 Medium (batch 4K, 8B tokens), MERIT reaches validation loss 2.982 versus 3.068 for LAMB and 3.172 for AdamW.
- MERIT enables a 6K batch on GPT-2 Medium with no degradation relative to AdamW at batch 480 over 48B tokens (zero-shot avg 46.70 vs 46.59).
- Element-wise trust ratio overhead is about 1% relative to LAMB (GPT-2 medium step time 58.50s vs 57.93s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MERIT instead of LAMB | GPT-2 Small (125M), large-batch training · OpenWebText, batch 1K | validation loss: 3.355 -> 3.280 | Helped |
| Use MERIT instead of LAMB | GPT-2 Medium (355M), large-batch training · OpenWebText, batch 4K | validation loss: 3.068 -> 2.982 | Helped |
| Use MERIT instead of LAMB | GPT-2 Large (770M), large-batch training · OpenWebText, batch 8K | validation loss: 2.971 -> 2.897 | Helped |
| Train with MERIT at batch 4k instead of AdamW at batch 480 | GPT-2 Small · SuperGLUE zero-shot | Avg: 43.56 -> 43.87 | Helped |
| Train with MERIT at batch 6k instead of AdamW at batch 480 | GPT-2 Medium · SuperGLUE zero-shot | Avg: 46.59 -> 46.70 | Helped |
| Remove element-wise update clipping (ablation) | MERIT · OpenWebText, GPT-2 large-batch training | convergence | Hurt |
| Remove the weight-wise ratio lower bound (ablation) | MERIT · OpenWebText, GPT-2 large-batch training | generalization performance | Hurt |
| Remove the element-wise trust ratio (ablation) | MERIT · OpenWebText, GPT-2 large-batch training | generalization capability | Hurt |
| Replace the weight-wise l2-norm with max-norm, omitting element-wise ratios and clipping (ablation) | LAMB (maxLAMB variant) · OpenWebText, GPT-2 Small | validation loss: 3.355 -> 3.304 | Helped |
| Replace the weight-wise l2-norm with max-norm, omitting element-wise ratios and clipping (ablation) | LAMB (maxLAMB variant) · OpenWebText, GPT-2 Medium | validation loss: 3.068 -> 3.040 | Helped |
Surprising findings
- QK-Norm allows larger learning rates for AdamW but degrades performance in large-batch training.
- maxLAMB (max-norm trust ratio alone) only marginally improves over LAMB, so gains are attributed to the combined element-wise ratio and clipping.
Limitations
- Evaluated only on small GPT-2 and Llama models up to 770M parameters; no large-scale or in-context learning evidence.
- Not evaluated outside language modeling (e.g., vision or RL).
- Downstream evaluation covers only a few zero-shot tasks.
Experimental setup
Datasets: OpenWebText C4
Benchmarks: OpenWebText validation loss LAMBADA WikiText SuperGLUE ARC COPA HellaSwag RACE WIC
Metrics: validation loss perplexity zero-shot accuracy top Hessian eigenvalue TFLOPS wall-clock time per step
Baselines: AdamW LAMB Lion Sophia-G maxLAMB QK-Norm
Builds on: LAMB AdamW Sophia Lion QK-Norm
Closest papers by meaning
- Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training (NeurIPS 2025 · similarity 0.88)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.88)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.88)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.87)
- Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures (ICLR 2025 · similarity 0.87)
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.