Research map / Architectures and efficiency / Quantization
Collage: Light-Weight Low-Precision Strategy for LLM Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Collage, a plugin that represents BF16 model weights and AdamW optimizer states as length-2 multi-component float expansions, enabling strict low-precision training without FP32 master weights, plus an effective descent quality metric to track lost arithmetic.
Problem
Low-precision (BF16) LLM training suffers from lost arithmetic and rounding errors in parameter and optimizer-state updates, and the standard fix (FP32 master weights) costs memory and speed.
Approach
Replace critical BF16 variables (parameters, second-moment states, beta2) with length-2 multi-component float expansions updated via Fast2Sum/Grow/Mul, so rounding errors are carried in a second low-precision component instead of upcasting to FP32; also introduce the effective descent quality metric to measure information lost at the update step.
Key results
- Collage-plus matches or exceeds FP32 master-weight mixed precision on BERT-base GLUE (average 0.8100 vs 0.8090) and RoBERTa-base GLUE (average 0.7986 vs 0.7901).
- Collage-light achieves up to 3.74x throughput speed-up over the FP32 master-weight baseline for GPT-6.7B, and peak memory savings average 23.8% (light) and 15.6% (plus) versus the baseline.
- On GPT-125M to 6.7B and OpenLLaMA-7B, Collage formations reach validation perplexity similar to FP32 master weights, while Collage-light underperforms at beta2=0.999 and Collage-plus stays close.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add BF16 MCF components to parameters and optimizer states | Collage-plus · BERT-base and BERT-large pretraining | perplexity: BERT-base Phase-1: 5.26 vs 5.26; BERT-base Phase-2: 4.66 vs 4.71; BERT-large Phase-1: 3.94 vs 4.06; BERT-large Phase-2: | Helped |
| Use Collage-plus rather than FP32 master weights | BERT-base and RoBERTa-base GLUE fine-tuning · GLUE average score | average score: BERT-base: 0.8090 -> 0.8100; RoBERTa-base: 0.7901 -> 0.7986 | Mixed |
| Use Collage-light rather than FP32 master weights | GPT models, β₂=0.95 · GPT-1.3B, GPT-2.7B, GPT-6.7B validation | perplexity: 8.50~|~17.70; 8.33~|~11.36; 8.17~|~11.13 | No clear effect |
| Use Collage-light rather than FP32 master weights | GPT training · GPT-6.7B | relative speedup: 3.74× | Helped |
| Use Collage-light rather than FP32 master weights | GPT and OpenLLaMA training · Peak GPU memory | peak memory saved: -0.8(-19.3%); -7.6(-21.5%); -15.6(-23.8%); -38.2(-26.6%); -49.2(-27.8%) | Helped |
| Use Collage-light rather than FP32 master weights | GPT-30B training · UBS=1, sequence length 2,048 | OOM-free run | Helped |
| Add MCF for optimizer states in addition to parameter MCF (ablation) | Collage-light · GPT-125M, β₂=0.999 | validation perplexity: 15.80 -> 15.18 (Global BatchSize=1024) | Helped |
| Add MCF for optimizer states in addition to parameter MCF (ablation) | Collage-light · OpenLLaMA-7B, β₂=0.99 | training perplexity: 8.00 -> 6.11 | Helped |
| Increase β₂ to 0.999 (ablation) | Collage-light · GPT-125M, Global BatchSize=1024 | validation perplexity: 15.03 -> 15.80 | Hurt |
| Increase β₂ to 0.999 (ablation) | Collage-plus · GPT-125M, Global BatchSize=1024 | validation perplexity: 15.03 -> 15.18 | Hurt |
| Increase global batch size from 1024 to 2048 (ablation) | Collage-light · GPT-125M, β₂=0.999 | validation perplexity: 15.80 -> 15.18 | Helped |
Surprising findings
- Simply giving the optimizer FP32 states without master weights (D^-MW) fails to match Collage-plus quality, showing that compensating floating-point errors matters more than raw precision.
- Collage-light closely matches the FP32 master-weight baseline for beta2=0.98 and 0.95/0.99 but lags at beta2=0.999 for BERT and GPT-125M.
Limitations
- Gains depend on beta2; Collage-light underperforms at beta2=0.999, requiring Collage-plus.
- Speed and memory results are measured on specific AWS p4.24xlarge and A100 setups with models up to 7B, and the 30B experiment is limited to memory feasibility.
- Authors note the optimal beta2 varies case by case and no clear conclusion relating beta2 to converged performance is established.
Experimental setup
Datasets: Wikipedia-en GLUE
Benchmarks: GLUE
Metrics: pre-training perplexity train perplexity validation perplexity GLUE accuracy throughput speed-up peak GPU memory effective descent quality
Baselines: BF16 parameters only BF16 + FP32 master weights (mixed precision) BF16-Kahan BF16 + FP32 optimizer states without master weights
Builds on: AdamW Kahan summation mixed-precision training multiple-component floating-point expansions
Closest papers by meaning
- Scaling Laws for Floating–Point Quantization Training (ICML 2025 · similarity 0.88)
- Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs (ICML 2024 · similarity 0.86)
- HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs (NeurIPS 2025 · similarity 0.85)
- Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.85)
- FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic (NeurIPS 2025 · similarity 0.85)
- Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.