Research map / Architectures and efficiency / Quantization
HOT: Hadamard-based Optimized Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HOT, a backpropagation optimization that applies Hadamard quantization to the activation-gradient path and Hadamard low-rank approximation plus INT8 quantization to the weight-gradient path, with activation buffer compression and layer-wise quantizer selection, reporting large memory savings and speedups on RTX 3090.
Problem
Reducing the memory and compute cost of backpropagation for matrix multiplications during fine-tuning and pretraining, while preserving training quality and achieving real (not only theoretical) speedups.
Approach
Selectively applies Hadamard-based optimizations according to the sensitivity of each backward path: INT4 Hadamard quantization for the activation gradient (gx) and Hadamard low-rank approximation combined with INT8 quantization for the weight gradient (gw), plus compressing stored activations at forward time and choosing per-token vs per-tensor quantization per layer.
Key results
- HOT reaches 77.31% on ImageNet-100 pretraining of ViT-B vs 77.87% FP and 75.97% LUQ, while LBP-WHT fails at 54.25%.
- HOT achieves up to 75% memory reduction in ViT models and up to 86% in ResNet-50 at batch size 256 versus FP.
- Per-layer BP latency on RTX 3090 shows HOT at 2.6x average speedup for ViT-B layers, up to 3.3x for fc2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply HOT instead of FP32 backpropagation | ViT-B · ImageNet-1k | accuracy: 70.01 -> 69.4 | Hurt |
| Apply HOT instead of FP32 backpropagation | ViT-B · ImageNet-100 | accuracy: 77.87 -> 77.31 | Hurt |
| Apply HOT instead of FP32 backpropagation | Llama3-8B · Alpaca | perplexity: 3.28 -> 3.29 | Hurt |
| Apply HOT instead of FP32 backpropagation | BERT-base · MRPC | accuracy: 86.52 -> 84.31 | Hurt |
| Use HOT instead of FP32 for backward computation | ViT-B · ViT-B qkv layer | latency: 182 -> 70 | Helped |
| Use HOT instead of FP32 for backward computation | ViT-B · ViT-B fc2 layer | latency: 233 -> 72 | Helped |
| Use 4-bit quantization with Hadamard transformation on the activation-gradient path instead of FP (ablation) | backpropagation on ResNet50 · CIFAR100 | accuracy: 76.46 -> 72.43 | Hurt |
| Use internal HLA on the activation-gradient path instead of FP (ablation) | backpropagation on ResNet50 · CIFAR100 | accuracy: 76.46 -> 76.29 | No clear effect |
| Use 4-bit quantization on the weight-gradient path instead of FP (ablation) | backpropagation on ResNet50 · CIFAR100 | accuracy: 76.46 -> 73.4 | Hurt |
| Use Hadamard transformation with 4-bit quantization on the weight-gradient path instead of FP (ablation) | backpropagation on ResNet50 · CIFAR100 | accuracy: 76.46 -> 76.16 | No clear effect |
| Use external HLA on the weight-gradient path instead of FP (ablation) | backpropagation on ResNet50 · CIFAR100 | accuracy: 76.46 -> 72.01 | Hurt |
| Use internal HLA on the weight-gradient path instead of FP (ablation) | backpropagation on ResNet50 · CIFAR100 | accuracy: 76.46 -> 51.10 | Hurt |
Surprising findings
- Internal and external Hadamard low-rank approximation cause large accuracy loss on the activation-gradient path, unlike the weight-gradient path, where HLA is robust but quantization is sensitive.
- LUQ and LBP-WHT fail to train (NaN) on some architectures such as EfficientNetV2 and Llama3-8B.
Limitations
- Performance degradation increases as model size and depth grow.
- LQS selection is a hand-crafted empirical procedure based on a calibration set and a 50% MSE-difference threshold.
Experimental setup
Datasets: ImageNet-1k ImageNet-100 CIFAR10 CIFAR100 COCO PASCAL VOC Cityscapes MRPC Alpaca
Benchmarks: CIFAR10 CIFAR100 ImageNet-1k ImageNet-100 PASCAL VOC Cityscapes MRPC Alpaca
Metrics: accuracy perplexity memory consumption (GB) latency (us) bit operations (TBOPs)
Baselines: LoRA LUQ LBP-WHT
Builds on: LBP-WHT LUQ LoRA Hadamard quantization
Closest papers by meaning
- HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs (NeurIPS 2025 · similarity 0.86)
- QiMeng-GEMM: Automatically Generating High-Performance Matrix Multiplication Code by Exploiting Large Language Models (AAAI 2025 · similarity 0.84)
- LoQT: Low-Rank Adapters for Quantized Pretraining (NeurIPS 2024 · similarity 0.84)
- FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic (NeurIPS 2025 · similarity 0.84)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.84)
- FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.