Research map / Architectures and efficiency / Attention and transformers
Learning to Compress Prompts with Gist Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use gist-token attention masking (single gist token) instead of TF-IDF discrete compression | LLaMA-7B · Alpaca+ Human | ChatGPT win rate vs positive control (%): 24.6 -> 45.8 | Helped |
| use gist token versus no-task negative control (ablation) | LLaMA-7B · Alpaca+ Human | ROUGE-L: 14.4 -> 23.9 | Helped |
| gist compression versus uncompressed prompt (positive control) on OOD prompts | FLAN-T5-XXL · Alpaca+ Human | ChatGPT win rate vs positive control (%): 50.0 -> 42.5 | Hurt |
| gist compression versus positive control on seen prompts | LLaMA-7B · Alpaca+ Seen | ROUGE-L: 58.0 -> 57.8 | No clear effect |
| increase number of gist tokens from 1 up to 10 (ablation) | Gist models · Alpaca+ validation splits | ROUGE-L / ChatGPT win rate | No clear effect |
| gist caching versus no caching | LLaMA-7B · Human split inference | GFLOPs: 915 -> 552 | Helped |
| gist caching versus no caching | FLAN-T5-XXL · Human split inference | CUDA time (ms): 31.0 -> 29.7 | Helped |
| gist caching versus instruction caching | LLaMA-7B · Human split inference | GFLOPs: 553 -> 552 | No clear effect |
| human annotators versus ChatGPT as judge | Gist models · Human split (100 examples) | win rate vs positive control (%): 52.3 vs 48.0 (LLaMA-7B); 40.6 vs 42.0 (FLAN-T5-XXL) | No clear effect |
Closest papers by meaning
- GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottlenecks (ICML 2024 · similarity 0.88)
- Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.87)
- Soft Prompt Recovers Compressed LLMs, Transferably (ICML 2024 · similarity 0.87)
- UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression (NeurIPS 2025 · similarity 0.87)
- Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models (NeurIPS 2024 · similarity 0.87)
- Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.