Research map / Architectures and efficiency / Pruning and efficient tuning
BackSlash: Rate Constrained Optimized Training of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BackSlash, a rate-distortion-optimized training method that adds a generalized-Gaussian rate penalty to the loss so LLM weights become compressible with exp-Golomb coding during training.
Problem
Model compression of LLMs is usually applied after training; the paper targets compressing parameters during training to trade off accuracy and model size for edge deployment.
Approach
Adds a discretized generalized-Gaussian rate term (sum of |theta|^nu with a dynamically estimated shape nu and soft gradient clipping) to the training loss, and entropy-codes the resulting parameter indices with exp-Golomb (k=0) codes after value mapping.
Key results
- On BERT sentiment analysis, BackSlash with EG coding reduces average code length from 7.31 to 2.64 bits (74% compression) with accuracy 92.59% vs 93.63% for normal training.
- Across BERT, GPT-2, Llama-3 and Gemma-2 on sentiment analysis, BackSlash yields 74%-89% EG compression with accuracy changes within about 1 point.
- Pruning experiments on BERT report the BackSlash model maintaining accuracy at 80% pruning rate, while normal training degrades at 50-60%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply rate-constrained BackSlash training | BERT · Sentiment analysis | EG code length: 7.31 -> 2.64 | Helped |
| Apply rate-constrained BackSlash training | GPT-2 · Sentiment analysis | accuracy: 85.92% -> 88.73% | Helped |
| Apply rate-constrained BackSlash training | Llama-3 · Sentiment analysis | accuracy: 86.09% -> 86.93% | Helped |
| Apply rate-constrained BackSlash training | Gemma-2 · Sentiment analysis | accuracy: 86.95% -> 85.86% | Hurt |
| Apply rate-constrained BackSlash training | BERT · SQuAD | next token accuracy: 99.97% -> 99.97% | No clear effect |
| Apply rate-constrained BackSlash training | BERT · WMT-19 | next token accuracy: 99.96% -> 99.95% | Hurt |
| Replace adaptive DGGR with L0.5 regularization (ablation) | BackSlash · BERT sentiment analysis | EG code length: 1.37 -> 2.90 | Hurt |
| Replace adaptive DGGR with L0.5 regularization (ablation) | BackSlash · BERT sentiment analysis | accuracy: 91.18% -> 91.88% | Helped |
| Replace adaptive DGGR with L1 regularization (ablation) | BackSlash · BERT sentiment analysis | accuracy: 91.18% -> 90.65% | Hurt |
| Replace adaptive DGGR with L2 regularization (ablation) | BackSlash · BERT sentiment analysis | accuracy: 91.18% -> 88.29% | Hurt |
| Replace adaptive DGGR with L2 regularization (ablation) | BackSlash · BERT sentiment analysis | EG code length: 1.37 -> 1.16 | Helped |
Surprising findings
- Accuracy was reported as non-monotonic in the Lagrange multiplier, suggesting an optimal intermediate value.
- Huffman coding gives only minimal gains over exp-Golomb codes on BackSlash-trained models.
Limitations
- The Lagrange multiplier is chosen by trial and error with no principled selection rule.
- Experiments are mostly classification on BERT; generation results are limited to next-token accuracy on DeepSeek with little detail.
- Hardware, training budget, and inference speed are not reported, so edge-deployment benefits are not measured.
Experimental setup
Datasets: IMDB Enron-Spam 20 Newsgroups SQuAD WMT-19
Benchmarks: IMDB Enron-Spam 20 Newsgroups SQuAD WMT-19
Metrics: accuracy average code length (bits) compression rate vs fixed-length coding next token accuracy
Baselines: unconstrained training with EG coding unconstrained training with Huffman coding DGGR vs L0.5 regularization L1 regularization L2 regularization
Builds on: rate-distortion optimization exp-Golomb codes Huffman coding
Closest papers by meaning
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.90)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.89)
- Language Modeling Is Compression (ICLR 2024 · similarity 0.89)
- EvoPress: Accurate Dynamic Model Compression via Evolutionary Search (ICML 2025 · similarity 0.89)
- Search for Efficient Large Language Models (NeurIPS 2024 · similarity 0.88)
- SliceGPT: Compress Large Language Models by Deleting Rows and Columns (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.