Research map / Architectures and efficiency / Quantization

GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling

NeurIPS 2026 · Architectures and efficiency › Quantization · approach: Post-training quantization of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GSQ, a post-training scalar quantizer that jointly learns per-coordinate grid assignments and group scales via Gumbel-Softmax relaxation (with local-shift for higher bits), closing much of the gap to VQ/trellis methods while staying kernel-compatible, including GGUF K-Quant refinement and trillion-parameter MoE quantization.

Problem

Scalar weight quantization for LLMs plateaus in accuracy at 3-4 bpp and falls well behind vector/trellis quantization at 2-3 bpp, while the latter are hard to implement and scale to very large MoE models.

Approach

Reformulate layer/block reconstruction as differentiable discrete assignment: each weight coordinate gets Gumbel-Softmax logits over the grid (or 5 local-shift logits for b>2) warm-started from GPTQ, jointly optimized with group scales using Lion, yielding symmetric group-wise scalar quantization compatible with existing kernels.

Key results

• On Llama-3.1-70B-Instruct at 2 bits, GSQ averages 75.57 vs. EfficientQAT 71.43 (best scalar baseline), trailing QTIP by 1.68 points.
• On Qwen3-8B GGUF, GSQ-Q2_K average improves from 50.03 (Unsloth) to 56.28.
• Kimi-K2.5 2-bit GSQ retains most reasoning performance (e.g., LiveCodeBench-v6 69.37 vs. base 61.37) while compressing from 4.5 to 2.13 bpp.

Surprising findings

• Lion optimizer outperforms AdamW by 3.70 average points for Gumbel-Softmax logits due to vanishing gradients in saturated relaxations.
• Kimi-K2.5 2-bit GSQ drops notably on GPQA Diamond (76.57 vs 89.29), attributed to calibration data composition.

Limitations

• Auxiliary logits impose memory overhead of 2-5x weight size, making full-model end-to-end optimization impractical.
• Depends on good initialization (GPTQ or GGUF); local-shift assumes assignments stay near initial grid point.
• Does not match VQ/trellis methods' expressiveness at fixed bit-width in all cases (e.g., 2-bit 70B still trails QTIP by 1.68 points).

Experimental setup

Datasets: FineWeb-Edu OpenThoughts OpenPlatypus

Benchmarks: ARC-Challenge ARC-Easy HellaSwag PIQA WinoGrande AIME25 GPQA Diamond MATH500 LiveCodeBench-v6 OpenAI-MRCR MMLU-Pro

Metrics: zero-shot accuracy pass@1 average generated tokens

Baselines: GPTQ AWQ QuIP EfficientQAT QTIP PV-Tuning Unsloth GGUF K-Quant

Builds on: GPTQ MaskLLM EfficientQAT QTIP Gumbel-Softmax

Closest papers by meaning

• Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment (NeurIPS 2025 · similarity 0.90)
• RSQ: Learning from Important Tokens Leads to Better Quantized LLMs (NeurIPS 2026 · similarity 0.90)
• ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.90)
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.90)
• LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid (ICLR 2025 · similarity 0.90)
• SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models (ICML 2025 · similarity 0.89)