Research map / Architectures and efficiency / Quantization

HESTIA: A Hessian-Guided Differentiable Quantization-Aware Training Framework for Extremely Low-Bit LLMs

NeurIPS 2026 · Architectures and efficiency › Quantization · approach: Post-training quantization of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HESTIA, a ternary QAT framework that replaces STE with a temperature-controlled softmax relaxation and uses offline Hutch++ Hessian-trace sensitivity to set per-tensor annealing rates, improving zero-shot accuracy on Llama-3.2 1B/3B.

Problem

Extremely low-bit (ternary, 1.58-bit) QAT for LLMs relies on the straight-through estimator, which creates a forward/backward mismatch and dead-zone optimization stagnation, and uses uniform annealing that ignores layer-wise sensitivity.

Approach

Relax ternary rounding into a softmax expectation over {-1,0,1} with temperature annealing, blend it with full-precision weights via a compress-stage pressure parameter, and modulate each tensor's temperature by a Hessian-trace-derived sensitivity score (repurposing the Hessian trace as a temporal scheduler rather than a bit-width allocator).

Key results

• On LLaMA-3.2-1B, HESTIA reaches 0.547 five-task average vs 0.519 for Tequila (10B tokens), a reported +5.39% relative gain.
• On LLaMA-3.2-3B, HESTIA reaches 0.601 average vs 0.576 for Tequila, a reported +4.34% gain.
• Ablation: removing Hessian-aware scheduling drops 1B average from 0.547 to 0.540 (group size 128).

Surprising findings

• The Hessian-aware schedule gives only +0.33% on 3B, indicating reduced benefit at larger scale.
• Group-wise quantization (group 128) alone contributes only a small gain (+0.56% on 1B), so most improvement is attributed to the scheduling.

Limitations

• Gains from Hessian scheduling are small on 3B (+0.33%) and the evaluation covers only five zero-shot tasks.
• Competitor results are cited from original papers rather than re-run, with the 1B/3B comparisons relying on Tequila's data setup.
• Mainly focuses on ternary quantization; Fairy2i cross-domain experiment is limited to Qwen2.5-0.5B with 6B tokens and small gains.

Experimental setup

Datasets: Ultra-FineWeb RedPajama

Benchmarks: ARC-Easy ARC-Challenge HellaSwag PIQA WinoGrande

Metrics: zero-shot accuracy average accuracy

Baselines: LSQ SEQ DLT TWN AbsMean Tequila INT2 LLM-QAT BitNet Spectra

Builds on: BitNet b1.58 straight-through estimator HAWQ Hutch++ DSQ Tequila

Closest papers by meaning

• QuEST: Stable Training of LLMs with 1-Bit Weights and Activations (ICML 2025 · similarity 0.90)
• BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.90)
• Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment (NeurIPS 2025 · similarity 0.89)
• LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025 · similarity 0.89)
• ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization (NeurIPS 2025 · similarity 0.89)
• LQER: Low-Rank Quantization Error Reconstruction for LLMs (ICML 2024 · similarity 0.89)