Research map / Architectures and efficiency / Quantization

HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models

NeurIPS 2026 · Architectures and efficiency › Quantization · approach: Post-training quantization of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Rollout-ResQ, a sparse residual correction on FP4 rollout activations, enabling end-to-end FP4 RL post-training with HiF4 (and MXFP4) on Qwen2.5 models while narrowing the gap to BF16.

Problem

End-to-end FP4 RL post-training (both rollout and training policies in 4-bit) degrades accuracy, and the dominant cause is rollout activation quantization where outliers cause underflow to zero.

Approach

Adds a 2:4 (or other) structured-sparse quantized residual of the activation quantization error as a second FP4 GEMM only in the rollout path, recovering outlier-induced loss at a fraction of dense-GEMM cost, combined with the HiF4 three-level scaling format.

Key results

• Rollout-ResQ-S2:4 with HiF4 on Qwen2.5-3B reduces the GSM8K gap to BF16 from 4.93 to 1.06 points (85.90 vs 86.96).
• With MXFP4, Rollout-ResQ-Dense/S2:4 narrows the GSM8K gap from 13.65 to 5.31 points.
• On Qwen2.5-Math-7B with HiF4, Rollout-ResQ-S2:4 narrows AIME24 gap from 11.14 to 6.56 points and Math500 gap from 15.19 to 5.64 points.

Surprising findings

• Keeping training in BF16 while quantizing rollout to FP4 performs worse than full FP4 training (78.01% HiF4, 50.57% MXFP4 vs 82.03%/73.31%).
• Quantizing the sparse residual rather than the primary activation is more robust to 2:4 sparsification.

Limitations

• All experiments use FP4 simulation, so real speedups are not measured.
• Only tested on Qwen2.5-3B and Qwen2.5-Math-7B with math tasks and limited dataset scale.
• Only 50% sparsity levels explored.

Experimental setup

Datasets: GSM8K DAPO-Math-17K Math-500

Benchmarks: GSM8K Math-500 AIME-2024 AIME-2025 AMC-2023

Metrics: accuracy (Mean@1) accuracy (Mean@32)

Baselines: BF16 HiF4 MXFP4 SmoothQuant RHT OCC

Builds on: GRPO HiFloat4 MXFP4 SmoothQuant VeRL

Closest papers by meaning

• ROLLVERIFY: BRIDGING EFFICIENCY AND ACCURACY IN LONG-TAIL ROLLOUT REINFORCEMENT LEARNING (NeurIPS 2026 · similarity 0.88)
• Optimizing Large Language Model Training Using FP4 Quantization (ICML 2025 · similarity 0.88)
• Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead (ICLR 2026 · similarity 0.87)
• RaZeR: Pushing the Limits of NVFP4 Quantization with Redundant Zero Remapping (NeurIPS 2026 · similarity 0.86)
• Scaling FP8 training to trillion-token LLMs (ICLR 2025 · similarity 0.86)
• DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training (NeurIPS 2026 · similarity 0.86)