Research map / Optimization / Optimizers

The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics

NeurIPS 2026 · Optimization › Optimizers · approach: Adam convergence and adaptive learning rates · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies tied-beta Adam (beta1=beta2) via a transformed ratio y_t that is approximately scale-stable, derives a scalar recursion for it, and uses a fixed 4-bit codebook to store it in place of the second moment, while linking Adam to Signum via a constant-ratio limit.

Problem

Adam's adaptive behavior is hard to interpret because of the interaction between its two EMAs; the paper asks what determines its adaptive denominator across scales and tasks, and whether the second-moment state can be compressed.

Approach

In the tied-beta regime, the second moment decomposes as v_t = m_t^2 (1 + beta/(1-beta) y_t), where y_t = (1-beta)/beta * (v_t - m_t^2)/m_t^2 is a transformed ratio with a stable, heavy-tailed distribution. y_t obeys a scalar recursion y_t = beta x_t^2 y_{t-1} + (1-x_t)^2 with x_t = m_{t-1}/m_t, so Adam can be run storing (m_t, y_t) and y_t quantized to a fixed 16-level codebook; a constant y_t re

Key results

• TR-Adam with 4-bit y_t (FP32 m_t) matches Adam in Llama-style 20M pre-training: validation loss 3.706 vs 3.694 at beta=0.90 and 3.672 vs 3.672 at beta=0.95.
• On Pythia-1B SFT, TR-Adam with 4-bit y_t slightly beats Adam at every beta (e.g., 1.071 vs 1.076 at beta=0.92).
• On ReMax RL training, TR-Adam improves validation reward over Adam at beta=0.90 (0.825 vs 0.788 with FP32 m_t).

Surprising findings

• The 4-bit transformed-ratio variant sometimes matches or slightly exceeds full Adam in post-training (SFT and RLHF), suggesting a mild regularization effect.
• A constant y_t around 3 (Signum-like limit) gives a learning-rate scale close to Adam's, whereas pure Signum needs a separate and beta-dependent learning-rate search.

Limitations

• Experiments limited to Transformer language models up to 1B parameters.
• Analysis of the stable distribution relies on a local Gaussian window assumption and small-shift approximations.
• Gains in post-training are small (often within about 0.01 loss) and the RLHF setup is limited to ReMax.

Experimental setup

Datasets: FineWeb UltraChat

Benchmarks: FineWeb validation loss UltraChat SFT validation loss ReMax validation reward

Metrics: validation loss validation reward learning-rate sensitivity

Baselines: Adam Signum FP8 first-moment Adam variant

Builds on: Adam Signum FP8 optimizer state quantization

Closest papers by meaning

• In Search of Adam’s Secret Sauce (NeurIPS 2025 · similarity 0.91)
• A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.91)
• On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent (ICLR 2025 · similarity 0.88)
• Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.87)
• Adam with model exponential moving average is effective for nonconvex optimization (NeurIPS 2024 · similarity 0.87)
• Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · similarity 0.87)