Research map / Large language models / Language, safety and interpretability
Learning to Compress: Unlocking the Potential of Large Language Models for Text Representation
Yeqin Zhang, Yizheng Zhao, Chen Hu, Binxing Jiao, Daxin Jiang, Ruihang Miao, Cam-Tu Nguyen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use knowledge-distillation (KL) loss in continuation pretext instead of NLL (ablation) | LLM2Comp pretraining · MTEB 14-task avg | average score: 46.22 -> 52.49 | Improved |
| use continuation+KD instead of reconstruction pretext (ablation) | LLM2Comp pretraining · MTEB 14-task avg | average score: 35.79 -> 52.49 | Improved |
| replace MNTP pretext with CTKD compression | LLM2Vec · MTEB 14-task avg (pretraining only) | average score: 46.87 -> 52.49 | Improved |
| KD loss instead of NLL: training stability (ablation) | LLM2Comp pretraining · MTEB 5-run std | std: 5.32 -> 1.37 | Improved |
| increase memory tokens from 8 to 16 (ablation) | LLM2CompKL · MTEB retrieval | retrieval score | Hurt |
| vary memory tokens within 1 to 8 (ablation) | LLM2CompKL · MTEB non-retrieval tasks | score | No clear effect |
| increase pretraining samples from 32k to 128k (ablation) | LLM2CompNLL · MTEB 14-task avg | average score: 42.95 -> 48.38 | Improved |
| pool only de-duplicated effective tokens (ablation) | LLM2CompNLL (128k) · MTEB 14-task avg | average score: 48.38 -> 51.09 | Improved |
| add unsupervised contrastive learning after CTKD pretraining (ablation) | LLM2CompKL · MTEB 14-task avg | average score: 52.49 -> 58.51 | Improved |
| add supervised contrastive learning after UCL (ablation) | LLM2CompKL · MTEB 14-task avg | average score: 58.51 -> 66.78 | Improved |
| use CTKD pretext instead of MNTP before UCL | LLM2Comp vs LLM2Vec · MTEB 14-task avg | average score: 57.82 -> 58.51 | Improved |
| use CTKD pretext instead of MNTP before SCL, with 0.36M vs 1.16M samples | LLM2Comp vs LLM2Vec · MTEB 14-task avg | average score: 66.11 -> 66.78 | Improved |
Closest papers by meaning
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.91)
- ARC-Encoder: learning compressed text representations for large language models (NeurIPS 2026 · similarity 0.90)
- zip2zip: Inference-Time Adaptive Tokenization via Online Compression (NeurIPS 2025 · similarity 0.90)
- Language Modeling Is Compression (ICLR 2024 · similarity 0.90)
- Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.90)
- Rep2Text: Decoding Full Text from a Single LLM Token Representation (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
