Research map / Large language models / Language, safety and interpretability
FOCUS & RePAIR: Mitigating Text Degeneration via Token-Level Guidance For Pruned Large Language Models
Junyoung Lee, Sehyeon Park, Shinhyoung Jang, Seonha Ryu, Hojeong Kim, Hyunsei Lee, Il hong Suh, Yeseong Kim
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| width pruning | Llama (unpruned) · Repetition rate (sampling) | repetition rate: 5.9% -> 12.4% | Hurt |
| replace two onset tokens with top-2 alternative | Llama · Unpruned model, 1,000 sequences | CREP: 5.9% -> 0.7% | Improved |
| add FOCUS reweighting to KD (ablation) | KD · WikiText-103 continuation (width-pruned 25%), Llama-3.1-8B | CREP (%): 7.3 -> 1.73 | Improved |
| add FOCUS to KD (perplexity) (ablation) | KD · WikiText-103 continuation (width-pruned 25%), Llama-3.1-8B | perplexity: 21.69 -> 22.32 | Hurt |
| add RePAIR to KD (ablation) | KD · WikiText-103 continuation (width-pruned 25%), Llama-3.1-8B | CREP (%): 7.3 -> 2.23 | Improved |
| add RePAIR to KD (MAUVE) (ablation) | KD · WikiText-103 continuation (width-pruned 25%), Llama-3.1-8B | MAUVE: 0.61 -> 0.68 | Improved |
| add FOCUS on top of RePAIR (ablation) | RePAIR · WikiText-103 continuation (width-pruned 25%), Llama-3.1-8B | CREP (%): 2.23 -> 0.57 | Improved |
| add FOCUS to DITTO (ablation) | DITTO · WikiText-103 continuation (width-pruned 25%), Llama-3.1-8B | CREP (%): 5.27 -> 0.5 | Improved |
| add FOCUS + RePAIR on Llama-2-13B (ablation) | KD · WikiText-103 continuation (width-pruned 25%), Llama-2-13B | CREP (%): 1.13 -> 0.00 | Improved |
| add RePAIR to KD on Self-Instruct (ablation) | KD · Self-Instruct (width-pruned 25%) | CREP (%): 1.97 -> 0.63 | Improved |
| use onset-level pairs (RePAIR) instead of sequence-level DPO (ablation) | CE baseline · WikiText-103, Llama-3.1-8B | unique 6-gram repetition (%): 3.45 -> 3.01 (unique 6-gram) | Improved |
| FOCUS vs ToDi distribution-shaping distillation (ablation) | KD · WikiText-103 continuation (width-pruned 25%), Llama-3.1-8B | CREP (%): 6.73 -> 1.73 | Improved |
Closest papers by meaning
- Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing (ICLR 2025 · similarity 0.88)
- Restoring Pruned Large Language Models via Lost Component Compensation (NeurIPS 2025 · similarity 0.88)
- Closing the Curious Case of Neural Text Degeneration (ICLR 2024 · similarity 0.88)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.87)
- Improving Open-Ended Text Generation via Adaptive Decoding (ICML 2024 · similarity 0.87)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
