Research map / Large language models / Language, safety and interpretability
CLLMs: Consistency Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes pretrained LLMs on Jacobi decoding trajectories with a global consistency loss plus an AR loss so they reach the fixed point in fewer iterations, giving 2.4-3.4x speedups without extra model components.
Problem
Vanilla Jacobi decoding for LLMs yields only marginal speedup over autoregressive decoding because models rarely predict more than one correct token per fixed-point iteration.
Approach
Train the LLM so that any intermediate state on a Jacobi trajectory maps directly to the fixed point (AR output), using a consistency loss inspired by consistency models in diffusion, combined with an AR loss to preserve quality; no architectural changes or auxiliary heads are required.
Key results
- CLLM-Deepseek-7B achieves 3.4x speedup with Jacobi decoding on Spider (69.3 vs 70.0 execution accuracy for fine-tuned baseline).
- CLLM-LLaMA2-7B achieves 2.4x speedup on ShareGPT with Jacobi decoding and MT-bench score 6.4, versus 1.0x for AR and 6.5 for the fine-tuned baseline.
- CLLM-LLaMA2-7B reaches 3.0x speedup on GSM8K with Jacobi decoding while accuracy drops from 59.1 to 56.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the target LLM with global consistency and AR losses, then use Jacobi decoding | LLaMA2-7B · GSM8K | speedup: 1.1× -> 3.0× | Helped |
| Fine-tune the target LLM with global consistency and AR losses, then use Jacobi decoding | Deepseek-Coder-7B-Instruct · Spider | speedup: 1.0× -> 3.4× | Helped |
| Fine-tune the target LLM with global consistency and AR losses, then use Jacobi decoding | Deepseek-Coder-7B-Instruct · Spider | execution accuracy: 70.0 -> 69.3 | Hurt |
| Fine-tune the target LLM with global consistency and AR losses, then use Jacobi decoding | LLaMA2-7B · ShareGPT (MT-Bench) | speedup: 1.1× -> 2.4× | Helped |
| Increase the AR-loss weight from 1 to 10 (ablation) | Global-consistency CLLM · GSM8K | accuracy: 51.3 -> 56.4 | Helped |
| Increase the AR-loss weight from 1 to 10 (ablation) | Global-consistency CLLM · GSM8K | speedup: 3.2× -> 3.0× | Hurt |
| Replace global consistency loss with local consistency loss (ablation) | CLLM with standard AR-loss weight · GSM8K | speedup: 3.2× -> 2.8× | Hurt |
| Replace global consistency loss with local consistency loss (ablation) | CLLM with standard AR-loss weight · GSM8K | accuracy: 51.3 -> 55.2 | Helped |
| Increase trajectory dataset size from 20k to 500k (ablation) | ShareGPT CLLM · ShareGPT (MT-Bench), 16-token sequence | speedup: 1.7× -> 2.7× | Helped |
| Increase trajectory dataset size from 20k to 500k (ablation) | ShareGPT CLLM · ShareGPT (MT-Bench) | MT-Bench score: 6.1 -> 6.4 | Helped |
Surprising findings
- Jacobi decoding on the fine-tuned (unmodified-training) baseline gives only 1.0-1.1x speedup, while CLLMs reach 2.4-3.4x.
- Global consistency loss outperforms local consistency loss in the reported ablation.
Limitations
- Speedup depends strongly on the quality and size of the Jacobi trajectory dataset; open-domain ShareGPT needs more data.
- Only greedy decoding is considered; other sampling strategies are left to future work.
- Accuracy drops slightly on some tasks (e.g., GSM8K 59.1 to 56.4) and dataset generation requires running the teacher.
Experimental setup
Datasets: Spider CodeSearchNet Python GSM8K ShareGPT WikiText2 PTB
Benchmarks: GSM8K Spider CodeSearchNet Python MT-bench HumanEval WikiText2 PTB
Metrics: accuracy (test@1) execution accuracy pass@1 MT-bench score speedup (wall-clock tokens/s) perplexity
Baselines: Jacobi decoding Medusa-2 speculative decoding with distilled LLaMA-160m lookahead decoding fine-tuned LLaMA2-7B
Builds on: Jacobi decoding consistency models speculative decoding lookahead decoding
Closest papers by meaning
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (ICML 2024 · similarity 0.89)
- Multi-Branch Self-Drafting for LLM Inference Acceleration (AAAI 2025 · similarity 0.89)
- Progressive Mixed-Precision Decoding for Efficient LLM Inference (ICLR 2025 · similarity 0.89)
- Efficient Test-Time Scaling via Hierarchical Search and Self-Verification for Discrete Diffusion Language Models (ICML 2026 · similarity 0.89)
- AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism (ICML 2025 · similarity 0.89)
- Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.