Research map / Optimization / Optimizers

LOTION: Smoothing the Optimization Landscape for Quantized Training

NeurIPS 2026 · Optimization › Optimizers · approach: Constrained and non-backprop training · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LOTION, which smooths the quantized training loss by replacing it with its expectation under unbiased randomized-rounding noise, adding a curvature-aware Gauss-Newton regularizer, and reports gains over QAT and PTQ on a synthetic regression task and 150M/300M LMs.

Problem

Training neural networks against quantized objectives is hard because the quantizer is piecewise constant, giving zero gradients almost everywhere; STE-based QAT lacks convergence guarantees and PTQ often underperforms at low bit-widths.

Approach

Replace the quantized loss with its expectation under unbiased randomized-rounding noise (a Nesterov/randomized-smoothing style loss surrogate), which is differentiable almost everywhere, preserves global minima under stochastic rounding, and yields a second-order ridge-like regularizer weighted by curvature and rounding variance.

Key results

• On a 150M-parameter LM at INT4 with RTN, LOTION reaches 3.276 validation cross-entropy versus 3.315 for QAT and 3.864 for PTQ (RTN).
• On a 300M-parameter LM at INT4 with RTN, LOTION reaches 3.103 versus 3.223 for QAT and 3.606 for PTQ.
• On the synthetic linear regression task at INT4, LOTION outperforms both QAT and PTQ in quantized validation loss, and continues decreasing where QAT plateaus.

Surprising findings

• RAT (randomized rounding in the forward pass) is theoretically unbiased for gradients on quadratic losses, implying it performs similarly to PTQ, and QAT is likely no better than PTQ there.
• PTQ with randomized rounding at INT4 performs very poorly on 150M models (4.686 loss), worse than round-to-nearest PTQ.

Limitations

• Smoothing with randomized rounding still leaves non-differentiable points at quantization bin boundaries.
• Practical LM version uses empirical Fisher approximation and does not differentiate through it, and needs a tuned regularization weight lambda.
• Only weight quantization is studied; activation quantization is left for future work.

Experimental setup

Datasets: C4 synthetic linear regression data (power-law Gaussian inputs, d=12000)

Benchmarks: C4 validation cross-entropy synthetic quadratic validation loss

Metrics: quantized validation loss cross-entropy under RTN and RR quantization

Baselines: QAT PTQ RAT NIPQ STE-based QAT

Builds on: straight-through estimator randomized smoothing Sophia Adam OLMo

Closest papers by meaning

• Compute-Optimal Quantization-Aware Training (ICLR 2026 · similarity 0.88)
• Robust Training of Neural Networks at Arbitrary Precision and Sparsity (ICLR 2026 · similarity 0.87)
• Fine-tuning Quantized Neural Networks with Zeroth-order Optimization (ICLR 2026 · similarity 0.87)
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (ICML 2023 · similarity 0.87)
• HESTIA: A Hessian-Guided Differentiable Quantization-Aware Training Framework for Extremely Low-Bit LLMs (NeurIPS 2026 · similarity 0.87)
• LoTA-QAF: Lossless Ternary Adaptation for Quantization-Aware Fine-Tuning (NeurIPS 2025 · similarity 0.86)