Research map / Large language models / Instruction and fine-tuning

BitDelta: Your Fine-Tune May Only Be Worth One Bit

NeurIPS 2024 · Large language models › Instruction and fine-tuning · approach: Fine-tuning generalization and delta compression · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Quantizes the weight delta between a fine-tuned LLM and its base model to 1 bit (sign matrix plus per-matrix scale, calibrated by logit distillation), enabling multi-tenant serving with one high-precision base and many 1-bit deltas.

Problem

Fine-tuned LLMs are expensive to store and to serve concurrently because each full fine-tune requires a full set of weights in GPU memory; low-rank post-hoc approximations of fine-tune deltas fail because full fine-tune deltas are high-rank.

Approach

Decompose fine-tuned weights into base weights plus a delta, binarize the delta to its sign matrix times a single high-precision scale (initialized to mean absolute value, minimizing L2 error), then distill only the scales against the fine-tuned model's logits on a small C4 calibration set.

Key results

• BitDelta compresses fine-tune deltas by 10.87x (Llama 2-7B) to 15.41x (Llama 2-70B) and keeps adjusted average within about 1 point of the fine-tuned baseline on Llama 2-7B Chat (59.88 vs 59.81).
• On Llama 2-70B Chat, BitDelta scores 69.14 adjusted average vs 68.82 baseline, and MT-Bench 7.06 vs 7.12.
• BitDelta-Initial (no scale distillation) degrades TruthfulQA sharply (e.g., 41.10 vs 45.32 on Llama 2-7B Chat), while distillation recovers most of the gap; SVD rank-16 and rank-128 low-rank approximations underperform BitDelta on GSM8K and MT-Bench.

Measured effects reported in the paper

Surprising findings

• Sign-only 1-bit quantization of the delta with a single scale per matrix already retains most fine-tune performance before any calibration.
• Low-rank SVD approximations of full fine-tune deltas fail to capture fine-tune information even at rank 128, which matches the memory footprint of BitDelta.

Limitations

• Embedding and LM head layers are not quantized due to tokenizer vocabulary inconsistencies, limiting compression.
• BitDelta underperforms S-LoRA kernel latency in large-batch settings.
• Lossy compression may remove alignment information (dealignment risk noted in impact statement).

Experimental setup

Datasets: C4 UltraChat MT-Bench

Benchmarks: MT-Bench ARC Challenge BBH HellaSwag TruthfulQA LAMBADA WinoGrande GSM8K

Metrics: accuracy MT-Bench score adjusted average compression factor decoding latency GPU memory

Baselines: SVD low-rank delta approximation 8-bit RTN GPTQ QuIP# S-LoRA naive multi-model serving

Builds on: LoRA GPTQ Punica S-LoRA BitBLAS

Closest papers by meaning

• BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments (ICLR 2025 · similarity 0.89)
• LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits (ICML 2025 · similarity 0.89)
• When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.88)
• Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models (NeurIPS 2024 · similarity 0.88)
• DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.88)
• Two-stage LLM Fine-tuning with Less Specialization and More Generalization (ICLR 2024 · similarity 0.88)