Research map / Architectures and efficiency / Quantization
HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HALO, a quantization-aware fine-tuning method that places Hadamard rotations in forward and backward passes so all large linear-layer matmuls run in INT8/FP6/FP8, with kernels and quantized FSDP communication.
Problem
Quantized training of LLMs is hard during fine-tuning of pretrained models because weights, activations, and output gradients contain large outliers, and all three matmuls per linear layer must be low precision without losing accuracy or speed.
Approach
Use right-hand-side Hadamard rotations on weights and activations in the forward pass and a left-hand-side Hadamard on output gradients in the backward pass (HALO-2), with tensor-wise quantization, reusing quantized rotated tensors for memory and communication savings.
Key results
- INT8 HALO-2 matches BF16 within about 1% on GSM8K (68.3 vs 69.3), ViGGO (93.8 vs 94.0), and SQL (80.1 vs 79.9) during full fine-tuning, while SwitchBack drops to 64.0 on GSM8K.
- End-to-end INT8 HALO-2 speedup of 1.41x on eight RTX 4090 GPUs for Llama full fine-tuning, with peak linear-layer speedup of 1.82x.
- FP6 HALO-1 reaches 66.5 on GSM8K versus 69.3 BF16, while SwitchBack and Jetfire reach about 62.7-62.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add right-hand Hadamard transforms in the forward pass (HALO-1 vs HALO-0) (ablation) | INT8 fine-tuning · GSM8K | accuracy: 4.50 ± 1.03 -> 62.27 ± 0.64 | Helped |
| add left-hand Hadamard transforms on errors (HALO-2 vs HALO-1) (ablation) | INT8 fine-tuning · GSM8K | accuracy: 62.27 ± 0.64 -> 68.15 ± 0.08 | Helped |
| add right-hand Hadamard transforms in the forward pass (HALO-1 vs HALO-0) (ablation) | FP6 fine-tuning · GSM8K | accuracy: 62.32 ± 0.65 -> 66.54 ± 0.22 | Helped |
| add left-hand Hadamard transforms on errors (HALO-2 vs HALO-1) (ablation) | FP6 fine-tuning · GSM8K | accuracy: 66.54 ± 0.22 -> 67.42 ± 0.99 | Helped |
| use HALO-2 INT8 instead of Jetfire | GSM8K fine-tuning of Qwen-14B · GSM8K | accuracy: 81.92 $\pm$ 1.12 -> 82.33 $\pm$ 0.68 | Helped |
| use HALO-2 INT8 instead of Jetfire | GSM8K fine-tuning of Qwen-32B · GSM8K | accuracy: 87.1 $\pm$ 0.71 -> 88.40 $\pm$ 0.8 | Helped |
| use INT8 HALO-2 instead of SwitchBack | full fine-tuning · GSM8K | accuracy: 64.0 ± 0.7 -> 68.3 ± 0.1 | Helped |
| use INT8 HALO-2 instead of Jetfire | full fine-tuning · ViGGO | accuracy: 93.6 ± 0.4 -> 93.8 ± 0.1 | No clear effect |
| use FP6 HALO-1 instead of Jetfire | full fine-tuning · GSM8K | accuracy: 62.8 ± 1.1 -> 66.5 ± 0.2 | Helped |
| add HQ-FSDP quantized communication | HALO training on four RTX 4090 GPUs · three Llama-small decoder blocks, BS=4 | speedup: 0.99$\times$ -> 1.37$\times$ | Helped |
| use INT8 HALO-2 instead of BF16 | full fine-tuning · ViGGO | accuracy: 94.0 ± 0.3 -> 93.8 ± 0.1 | No clear effect |
| use FP8 without Hadamard transforms instead of BF16 | full fine-tuning · GSM8K | accuracy: 69.3 ± 0.5 -> 69.2 ± 0.2 | No clear effect |
Surprising findings
- Tensor-wise quantization with Hadamard rotations matches the more complex block-wise Jetfire scheme on INT8 fine-tuning.
- Pre-training TinyLlama-1.1B with INT8 and FP4 diverges regardless of HALO level, while FP6 converges only at level 2.
Limitations
- Speedups rely on RTX 4090 GPUs; FP6 hardware matmul unsupported so FP8 used as lower bound.
- Fine-tuning evaluated for single epoch with only three datasets; pre-training results show INT8 and FP4 diverge.
- Uses tensor-wise RTN quantization; FP6 still shows about 2.8% GSM8K drop.
Experimental setup
Datasets: ViGGO GSM8K SQL generation dataset C4
Benchmarks: GSM8K ViGGO SQL generation
Metrics: accuracy end-to-end speedup layer-wise speedup
Baselines: BF16 SwitchBack Jetfire LoRA No-HALO FP8
Builds on: QuaRot SwitchBack Jetfire LoRA FSDP
Closest papers by meaning
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 · similarity 0.89)
- ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization (NeurIPS 2024 · similarity 0.88)
- QuEST: Stable Training of LLMs with 1-Bit Weights and Activations (ICML 2025 · similarity 0.88)
- Optimizing Large Language Model Training Using FP4 Quantization (ICML 2025 · similarity 0.88)
- SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models (ICML 2025 · similarity 0.88)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.