Research map / Architectures and efficiency / Quantization
Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BinaryMoS, a 1-bit LLM binarization method whose scaling factors are multiple experts linearly combined per token by a learned router, trained with quantization-aware knowledge distillation.
Problem
Binarizing LLM weights to 1 bit greatly degrades linguistic performance, and prior LLM binarization methods either add high memory overhead or still lose much accuracy.
Approach
Replace static per-dimension scaling factors of binarized weights with multiple scaling experts for input and output dimensions, which a softmax router merges per token into token-adaptive scales, adding only a small memory overhead compared with MoE over full weight matrices.
Key results
- On LLaMA-1-7B, BinaryMoS (1-bit) reaches 7.97 WikiText2 perplexity and 54.48 average zero-shot accuracy, versus 8.48 and 52.36 for OneBit and 64.06 for Float16.
- BinaryMoS outperforms 2-bit GPTQ and OmniQuant on WikiText2 perplexity across OPT and LLaMA models, e.g., 7.08 vs 7.84 for OmniQuant on LLaMA-1-13B.
- BinaryMoS achieves 9.65x to 11.24x memory compression versus Float16, close to OneBit's compression, and latency comparable to OneBit.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use token-adaptive scaling experts instead of OneBit's static scaling | 1-bit LLM binarization · WikiText2, OPT-125M; OPT-1.3B; LLaMA-1-7B; LLaMA-1-13B; LLaMA-2-7B; LLaMA-2-13B | perplexity: 39.45 -> 36.46; 20.36 -> 18.45; 8.48 -> 7.97; 7.65 -> 7.16; 8.60 -> 7.88; 7.56 -> 7.08 | Helped |
| Use token-adaptive scaling experts instead of OneBit's static scaling | 1-bit LLM binarization · C4, OPT-125M; OPT-1.3B; LLaMA-1-7B; LLaMA-1-13B; LLaMA-2-7B; LLaMA-2-13B | perplexity: 35.58 -> 33.13; 20.76 -> 18.83; 10.49 -> 9.72; 9.56 -> 8.81; 10.74 -> 9.75; 9.67 -> 8.91 | Helped |
| Use token-adaptive scaling experts instead of OneBit's static scaling | 1-bit LLM binarization · Common Sense tasks average, OPT-125M; OPT-1.3B; LLaMA-1-7B; LLaMA-1-13B; LLaMA-2-7B; LLaMA-2-13B | zero-shot accuracy: 42.84 -> 43.37; 47.50 -> 49.34; 52.36 -> 54.48; 55.10 -> 56.68; 52.42 -> 54.01; 55.29 -> 57.09 | Helped |
| Increase scaling experts from 1 to 2 (ablation) | BinaryMoS · LLaMA-1-7B quick assessment, WikiText2 | perplexity: 9.33 -> 9.19 | Helped |
| Increase scaling experts from 2 to 4 (ablation) | BinaryMoS · LLaMA-1-7B quick assessment, WikiText2 | perplexity: 9.19 -> 8.92 | Helped |
| Increase scaling experts from 2 to 4 (ablation) | BinaryMoS · LLaMA-1-7B quick assessment, C4 | perplexity: 12.18 -> 11.85 | Helped |
| Increase scaling experts from 4 to 8 (ablation) | BinaryMoS · LLaMA-1-7B quick assessment | perplexity: 8.92 -> 9.17; 11.85 -> 12.28 | Hurt |
| Train on WikiText2 only rather than the mixed WikiText2 and C4 dataset (ablation) | BinaryMoS, LLaMA-1-7B · C4 | perplexity: 11.85 -> 28.61 | Hurt |
| Train on WikiText2 only rather than the mixed WikiText2 and C4 dataset (ablation) | BinaryMoS, LLaMA-1-7B · Common Sense tasks average | zero-shot accuracy: 50.61 -> 45.32 | Hurt |
| Train on C4 only rather than the mixed WikiText2 and C4 dataset (ablation) | BinaryMoS, LLaMA-1-7B · WikiText2 | perplexity: 8.92 -> 13.76 | Hurt |
| Train on generated data rather than the mixed WikiText2 and C4 dataset (ablation) | BinaryMoS, LLaMA-1-7B · Common Sense tasks average | zero-shot accuracy: 50.61 -> 49.42 | Hurt |
| Use 4 scaling experts instead of 1 (ablation) | BinaryMoS · LLaMA-1-7B quick assessment, Common Sense tasks average | zero-shot accuracy: 48.89 -> 50.61 | Helped |
Surprising findings
- Increasing the number of scaling experts from 4 to 8 degraded performance, which the authors attribute to difficulty training the router.
- Training on WikiText2 alone caused overfitting and poor generalization to C4 and zero-shot tasks.
Limitations
- Substantial accuracy gap to Float16 remains after binarization.
- Training cost of QAT makes application to very large models such as LLaMA-2-70B challenging.
- Router training becomes harder as the number of scaling experts grows, and advanced MoE training techniques were not used.
Experimental setup
Datasets: WikiText2 C4
Benchmarks: WikiText2 C4 BoolQ PIQA HellaSwag WinoGrande ARC-e ARC-c
Metrics: perplexity zero-shot accuracy memory footprint latency
Baselines: PB-LLM BiLLM OneBit GPTQ OmniQuant
Builds on: OneBit BiLLM PB-LLM Mixture of Experts knowledge distillation
Closest papers by meaning
- ARB-LLM: Alternating Refined Binarizations for Large Language Models (ICLR 2025 · similarity 0.90)
- PB-LLM: Partially Binarized Large Language Models (ICLR 2024 · similarity 0.89)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.88)
- Treasures in Discarded Weights for LLM Quantization (AAAI 2025 · similarity 0.86)
- Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment (AAAI 2025 · similarity 0.86)
- BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.