Research map / Large language models / Language, safety and interpretability
Controlled Text Generation via Language Model Arithmetic
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces model arithmetic, a framework that composes and biases LLMs via formulas over output distributions (weighted-KL optimal combinations, a union operator), and generalizes speculative sampling to reduce model calls.
Problem
Controlled text generation lacks expressive, interpretable, and efficient ways to combine multiple attributes and to steer LLMs away from attributes without retraining or task-specific data.
Approach
Define output distributions as weighted-KL optima over input distributions, yielding a formula algebra (linear combinations, a max-based union operator, classifier terms) and extend speculative sampling so expensive formula terms are evaluated lazily via sub-formulas as proposal models.
Key results
- On /pol/ toxicity with Llama-2-13b, M - 0.96·Union(M_toxic, M) reaches toxicity 0.201 vs 0.288 unconditioned and 0.208 for PreAdd, with lower perplexity (11.08 vs 12.73).
- Speculative sampling of model arithmetic formulas reduces model calls per token to at most 1.44 (up to 64% reduction) and speeds inference up to 2.24x.
- GPT-4 prefers the union-based formula over PreAdd on toxicity reduction in most settings (e.g., 0.40 win vs 0.38 lose on Llama-2-13b).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the union operator for toxicity reduction instead of PreAdd | Llama-2-13b model arithmetic · /pol/ | toxicity: 0.20751357989225 -> 0.2014622639518 | Helped |
| Add a toxicity classifier term to union-based toxicity reduction (ablation) | Llama-2-13b model arithmetic · /pol/ | toxicity: 0.2014622639518 -> 0.1623 | Helped |
| Add a toxicity classifier term to union-based toxicity reduction (ablation) | Pythia-12b model arithmetic · /pol/ | toxicity: 0.1702074087756 -> 0.1478178452382 | Helped |
| Add a toxicity classifier term to union-based toxicity reduction (ablation) | MPT-7b model arithmetic · /pol/ | toxicity: 0.19345939548285 -> 0.1737395688082 | Helped |
| Increase union bias strength from 0.96 to 0.99 (ablation) | Model arithmetic toxicity reduction · /pol/ | toxicity: 0.2014622639518 -> 0.18614473078765; 0.1702074087756 -> 0.17138050593874998; 0.19345939548285 -> 0.18750884556234998 | Mixed |
| Increase union bias strength from 0.96 to 0.99 (ablation) | Model arithmetic toxicity reduction · /pol/ | perplexity: 11.078353402312155 -> 12.053630436852684 | Hurt |
| Add a classifier term to union-based sentiment control (ablation) | Llama-2-13b model arithmetic · IMDB negative reviews to positive | sentiment: 0.39331756591796874 -> 0.46628311157226565 | Helped |
| Add a classifier term to union-based sentiment control (ablation) | Llama-2-13b model arithmetic · IMDB positive reviews to negative | negative sentiment: 0.4691995391845703 -> 0.5239201202392578 | Helped |
| Use speculative sampling for model arithmetic | Llama-2-13b-Chat with F1 · Speculative sampling, F1 | time per token [ms]: 97.0255337245698 -> 43.25989481778973 | Helped |
| Use speculative sampling for model arithmetic | Llama-2-13b-Chat with F1 · Speculative sampling, F1 | calls per token: 4.0 -> 1.3165171374500595 | Helped |
| Increase the coefficient λ in M + λM_a (ablation) | Model arithmetic speculative sampling · M + λM_a | model calls per token | Hurt |
| Use the union operator together with a classifier rather than union alone (ablation) | Model arithmetic sentiment control · IMDB negative reviews to positive | perplexity: 10.170443906469423 -> 10.398901188444746 | Mixed |
Surprising findings
- Linear biasing with a higher strength (0.6 for PreAdd) already degrades perplexity more than the union operator at 0.96, showing the union operator permits stronger biasing without fluency loss.
- Sports attribute in the control experiments does not follow a linear curve; it only gets a significant boost once its relative strength exceeds 1.0.
Limitations
- Classifier terms require top-k approximation over the most likely tokens, restricting them to top-k sampling.
- Speculative speedup degrades as bias strength grows because KL divergence between proposal and target increases, lowering acceptance.
- Highest tested biasing strengths eventually degrade perplexity.
Experimental setup
Datasets: /pol/ dataset Jigsaw toxicity dataset Alpaca IMDB movie reviews
Benchmarks: /pol/ toxicity IMDB sentiment control Alpaca attribute control
Metrics: Perspective API toxicity score sentiment classifier score perplexity GPT-4 win/lose/draw preference model calls per token time per token
Baselines: SelfDebias FUDGE PreAdd GPT-4 preference judging
Builds on: speculative sampling FUDGE PreAdd SelfDebias
Closest papers by meaning
- Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs (ICLR 2025 · similarity 0.88)
- Successor Features for Efficient Multi-Subject Controlled Text Generation (ICML 2024 · similarity 0.88)
- LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (ICLR 2025 · similarity 0.88)
- Precise Information Control in Long-Form Text Generation (NeurIPS 2025 · similarity 0.87)
- Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo (ICLR 2025 · similarity 0.87)
- Language Generation with Strictly Proper Scoring Rules (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.