Research map / Large language models / Reasoning and chain of thought
Vision-centric Token Compression in Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vist, a slow-fast LLM context compression method that renders distant text into images encoded by a frozen CLIP ViT-L/14 and compressed by a trainable Resampler, trained with a frequency-masked contrastive alignment objective (PVE).
Problem
Long-context inputs strain LLM compute and memory; existing text-encoder compression (e.g., CEPE) keeps redundant tokens, so efficient long-context handling remains costly.
Approach
Render distant context as 224x224 text images processed by a frozen vision encoder and Resampler (fast path), while nearby tokens go to the LLM (slow path); PVE aligns visual features with text-token embeddings after frequency-based masking of high-frequency tokens.
Key results
- Vist reaches 46.3 average ICL accuracy vs 44.1 for CEPE* (n_e=18, n_d=2) across 11 datasets, and 50.4 vs 46.8 with 48 encoder demos.
- On open-domain QA with k_e=20, Vist scores 25.67 EM on TriviaQA vs 16.56 for CEPE*.
- Vist uses 2.3x fewer visual tokens than text tokens, with 0.87 GB lower memory than ToMe/FastV at 6144 tokens setting and reported ~16% FLOPs reduction relative to CEPE* at 16k tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Vist instead of CEPE* for encoder-based context compression | TinyLlama · ICL, 11 datasets, n_e=18 and n_d=2 | average accuracy: 44.1 -> 46.3 | Helped |
| Use Vist instead of CEPE* for encoder-based context compression | TinyLlama · ICL, 11 datasets, n_e=48 and n_d=2 | average accuracy: 46.8 -> 50.4 | Helped |
| Use Vist instead of CEPE* for context compression | TinyLlama · TriviaQA, k_e=5, k_d=10 | exact match: 16.41 -> 25.20 | Helped |
| Use Vist instead of CEPE* for context compression | TinyLlama · NQ, k_e=5, k_d=10 | exact match: 6.09 -> 8.71 | Helped |
| Use Vist instead of CEPE* for context compression | TinyLlama · PopQA, k_e=5, k_d=10 | exact match: 4.92 -> 11.44 | Helped |
| Add frequency-based masking to PVE (ablation) | Vist · ICL and open-domain QA | task performance | Helped |
| Replace frequency-based masking with random masking (ablation) | Vist · ICL and open-domain QA | task performance | Mixed |
| Vary the number of compressed visual tokens per image (ablation) | Vist · ICL and open-domain QA | task performance | Mixed |
| Increase encoder training input length (ablation) | Vist · Open-domain QA | exact match | Helped |
| Increase encoder training input length to 4096 tokens (ablation) | Vist · SST5 | accuracy: 42.7 | Helped |
| Increase encoder training input length to 6144 tokens (ablation) | Vist · MR | accuracy: 90.5 | Helped |
| Train with 1024 rather than 2048 encoder input tokens (ablation) | Vist · ICL | task performance | No clear effect |
Surprising findings
- Lower compression ratio (more visual tokens) does not always yield better results; 64 tokens per image beat 128 on most datasets.
- Using only the vision fast path (no LLM passages) on open-domain QA performs on par with TinyLlama given 10 passages.
Limitations
- Evaluated only on English and small models (TinyLlama, one extra 7B model).
- Rendering text into images and using ICL/QA on short-form tasks; benefit on tasks beyond text classification and QA is untested (authors note broader evaluation as future work).
- Compression ratio and performance trade-offs are dataset-dependent; some tasks (NLUS, TREC, TREF) underperform TinyLlama with full context.
Experimental setup
Datasets: RedPajama PG19 Proof Code SST2 MR AGN SST5 NLUS NLUI TREC TREF
Benchmarks: PG19 Proof Code ArXiv Book SST2 MR AGN SST5 NLUS NLUI TREC
Metrics: perplexity accuracy exact match TFLOPs memory (GB) throughput
Baselines: Replug Stream CEPE ToMe FastV TinyLlama
Builds on: CEPE Perceiver Resampler Flamingo-style cross-attention Pixel TinyLlama
Closest papers by meaning
- CORE: Compact Object-centric REpresentations as a New Paradigm for Token Merging in LVLMs (CVPR 2026 · similarity 0.91)
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.90)
- Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters (ICLR 2025 · similarity 0.90)
- EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling (CVPR 2026 · similarity 0.90)
- Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.89)
- ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.