Research map / Architectures and efficiency / Quantization

Normalized Architectures are Natively 4-Bit

NeurIPS 2026 · Architectures and efficiency › Quantization · approach: Post-training quantization of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that the normalized transformer nGPT is natively robust to NVFP4 4-bit training, stably training without Hadamard transforms or per-tensor scaling, and attributes this to coherent signal accumulation in dot products rather than lower quantization noise.

Problem

Low-bit (NVFP4) LLM training is fragile and standard transformers need overhead-heavy fixes (random Hadamard transforms, dynamic per-tensor scaling, mixed precision) to avoid divergence. The paper asks whether an architecture can be intrinsically robust to 4-bit quantization.

Approach

Uses nGPT, which constrains weights and hidden states to the unit hypersphere, and shows that this makes dot products accumulate quantized signal coherently (weak positive correlations among element-wise products), raising dot-product SNR and flattening the loss landscape, so NVFP4 training works without RHT or per-tensor scaling.</220>

Key results

• On a 1.2B dense model trained on 1T tokens, nGPT NVFP4 without RHT/scaling reaches loss 1.50 vs 1.52 for standard NVFP4 and GSM8K 35.56 vs 29.49.
• Dot-product SNR under NVFP4 on a 3.6B model is about 26 dB for nGPT vs 18.6 dB for GPT; signal correlation is about 14x larger in nGPT (1.32e-3 vs 9.31e-5).
• Single-layer GB200 benchmark shows the optimized nGPT NVFP4 path reaches roughly 3.3-3.6x speedup over the BF16 GPT baseline at the largest hidden sizes.

Surprising findings

• nGPT's optimal NVFP4 learning rate matches its BF16 optimum, whereas GPT's NVFP4 optimum is 16x larger than its BF16 optimum.
• Quantization noise correlation is near zero in both architectures, so the robustness gap is attributed entirely to signal correlation rather than noise reduction.

Limitations

• Mechanistic analysis uses a 3.6B nanoGPT-style model at D=4096; the source of the positive correlation is left for future work.
• The 3B/30B MoE trained for ~500B tokens, a short horizon compared to production models.
• Scaling predictions to D=16384 are theoretical extrapolations from the SNR model.

Experimental setup

Datasets: Nemotron-Pretraining-Datasets

Benchmarks: HellaSwag PIQA Winogrande MMLU MMLU Pro GSM8K

Metrics: validation loss relative error downstream accuracy dot-product SNR validation bits-per-byte speedup

Baselines: GPT-NVFP4 with RHT and per-tensor scaling BF16 GPT NVIDIA NVFP4 pretraining recipe

Builds on: nGPT NVFP4 pretraining recipe nanoGPT

Closest papers by meaning

• RaZeR: Pushing the Limits of NVFP4 Quantization with Redundant Zero Remapping (NeurIPS 2026 · similarity 0.88)
• Optimizing Large Language Model Training Using FP4 Quantization (ICML 2025 · similarity 0.88)
• any4: Learned 4-bit Numeric Representation for LLMs (ICML 2025 · similarity 0.88)
• QuEST: Stable Training of LLMs with 1-Bit Weights and Activations (ICML 2025 · similarity 0.87)
• ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs (NeurIPS 2026 · similarity 0.87)
• The Quantization Benefits of Residual-Free Transformers (NeurIPS 2026 · similarity 0.87)