Research map / Large language models / Reasoning and chain of thought

ReSET: Accurate Latency-Critical NVFP4 Reasoning via Step-Aware Temperature Scaling

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: KV cache compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ReSET, a step-aware entropy-based decoding temperature scaling method for NVFP4-quantized large reasoning models, plus a CUDA-core small-M NVFP4 GEMV kernel for latency-critical decoding.

Problem

NVFP4 quantization of large reasoning models degrades reasoning accuracy at symbolic tokens, and NVFP4 Tensor-Core kernels fail to realize latency gains at the small batch sizes (M≤8) used in latency-critical autoregressive decoding.

Approach

Token entropy is dominated by surrounding step-level uncertainty, so decoding temperature is adapted using a step-aware threshold (global threshold in confident steps, step-relative in uncertain ones) estimated online with a hybrid step-entropy estimator; paired with a CUDA-core NVFP4 GEMV kernel that avoids the 128-row Tensor-Core tile.

Key results

• ReSET improves NVFP4 AIME-120 average accuracy from 59.9 (RTN) to 62.5 across five models, vs. BF16 65.1.
• CUDA-core NVFP4 GEMV achieves 1.57-2.49x kernel-level latency reduction over vLLM-CUTLASS at M=1 on B200 across 12 Qwen3 projection shapes.
• End-to-end decoding reaches 1.97x speedup over BF16 on Qwen3-32B at batch size 1.

Surprising findings

• Quantization compresses entropy of high-entropy discourse tokens (e.g., 'Alternatively') in uncertain steps, so raising high-temperature helps rather than the opposite.
• Advanced PTQ methods (rotation, GPTQ) give only marginal perplexity gains over RTN under NVFP4, and adding ReSET on GPTQ yields little further benefit.

Limitations

• Kernel uses CUDA cores with FP16 FMA after dequantization rather than native FP4 arithmetic, so gains are limited to small-M decode.
• Calibration of the global threshold tau0 and T_low requires a small held-out calibration split per model/task.
• Gains on AIME-120 are around 2-3 points and some per-model results are mixed (e.g., Qwen3-8B GPQA, ReSET vs. RTN gap small).

Experimental setup

Datasets: NuminaMath-1.5 C4 FineWeb-Edu Wikitext-2 AIME 2022-2024 traces

Benchmarks: AIME-120 GPQA-Diamond LiveCodeBench Wikitext-2

Metrics: accuracy (averaged over 8 seeds) perplexity kernel latency (us) TPOT end-to-end decoding speedup

Baselines: RTN BRQ 4/6 MR-GPTQ BF16 baseline vLLM-CUTLASS top-p sampling min-p sampling

Builds on: NVFP4 RTN MR-GPTQ BRQ vLLM

Closest papers by meaning

• LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Loss (NeurIPS 2026 · similarity 0.88)
• SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning (NeurIPS 2025 · similarity 0.86)
• When Reasoning Meets Compression: Understanding the Effects of LLMs Compression on Large Reasoning Models (ICLR 2026 · similarity 0.86)
• R-KV: Redundancy-aware KV Cache Compression for Reasoning Models (NeurIPS 2025 · similarity 0.85)
• BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference (ICML 2026 · similarity 0.85)
• Entropy-Gated Latent Recursion (NeurIPS 2026 · similarity 0.85)