Research map / Vision-language and multimodal / Multimodal LLMs
VoCo-LLaMA: Towards Vision Compression with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VoCo-LLaMA, which uses the LLM itself to compress image tokens into a few VoCo tokens via attention masking and attention distillation during instruction tuning, extended to video via per-frame compressed token sequences.
Problem
VLMs are bottlenecked by the context window and compute cost of many vision tokens; prior compression uses external modules (Q-Former, pooling) that lose visual information at high compression ratios.
Approach
Insert VoCo tokens after the vision tokens and mask text-to-vision attention so text sees only VoCo tokens, letting the LLM itself distill how it processes vision tokens into the VoCo token activations, which can be cached and reused; then continue training on per-frame VoCo caches for video.
Key results
- Compressing 576 vision tokens into 1 VoCo token retains 83.7% of upper-bound performance averaged across seven image benchmarks.
- With 128 VoCo tokens, average retention reaches 97.7%.
- Versus no caching, VoCo-LLaMA cuts CUDA time by 69.6% and FLOPs by 94.8%; cache storage drops 99.8% relative to full caching.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compress 576 vision tokens into a single VoCo token via LLM attention distillation (vs. Q-Former/avg pooling) | Vicuna-7B VLM · 7 image benchmarks | compression retention rate: 83.7% vs 64.1% (avg pool+linear) and 57.2% (Q-Former) | Helped |
| Increase number of VoCo tokens from 1 to 128 (ablation) | VoCo-LLaMA · 7 image benchmarks | compression retention rate: 83.8% at 1 token to 97.7% at 128 tokens | Helped |
| Use 2 VoCo tokens per frame instead of more for video (per-frame compression with temporal modeling) | VoCo-LLaMA video · MSVD-QA / MSRVTT-QA / ActivityNet-QA | accuracy: 72.3 / 61.1 / 47.9 vs LLaMA-VID 69.7 / 57.7 / 47.4 | Helped |
| Increase VoCo tokens per frame from 2 to 8 for video (ablation) | VoCo-LLaMA video · MSVD-QA, MSRVTT-QA, ActivityNet-QA | accuracy: MSVD 72.3 to 73.4 (224 res); MSRVTT 61.1 to 62.0; ActivityNet 47.9 to 48.5 | Helped |
| Compress 576 tokens into 1 VoCo token with KV caching of VoCo activations | VoCo-LLaMA inference · single image inference, A100 | CUDA time and FLOPs vs no-cache baseline: CUDA time 440.5 ms to 134.0 ms (-69.6%); FLOPs 9.6T to 0.5T (-94.8%) | Helped |
| Cache 1 VoCo token instead of full 576-token KV cache | VoCo-LLaMA inference · single image inference | cache storage memory: 302.4 MB to 0.525 MB (99.8% reduction) | Helped |
| Replace 576-token full caching with 1 VoCo token: CUDA time | VoCo-LLaMA inference · single image inference | CUDA time (ms): 154.9 ms (full caching) to 134.0 ms | Helped |
| Train with 8 VoCo tokens on fine-grained REC/REG tasks instead of 1 (ablation) | VoCo-LLaMA fine-grained · REC / REG average retention | compression retention rate: REC 79.9% to 90.7%; REG 81.6% to 91.3% | Helped |
| Attention-based isolation of text tokens from vision tokens (VoCo design) vs random compression lower bound | VoCo-LLaMA · 7 image benchmarks | accuracy vs lower bound: consistently outperforms lower bound across all benchmarks (e.g. GQA 57.0 vs 37.7) | Helped |
| Use 32 VoCo tokens in training and vary inference token count (ablation) | VoCo-LLaMA · MMB/GQA/VQAv2/SEED | accuracy: Accuracy at 32 train-tokens, 16 inference tokens is lower than 32, but 4-token inference (MMB 59.7) beats 16 (58.3) | Mixed |
Surprising findings
- With 128 VoCo tokens, SEED-Bench performance exceeds the upper-bound model.
Limitations
- Single-token compression shows notable information loss (83.7% retention), with fine-grained REC/REG retention around 80%.
- Video modeling relies on continued training from image compression model and only 2 or 8 tokens per frame.
Experimental setup
Datasets: LLaVA-filtered CC3M WebVid Video-ChatGPT QA pairs GQA MMBench MME POPE SEED-Bench ScienceQA VQAv2 MSVD-QA MSRVTT-QA
Benchmarks: GQA MMBench MME POPE SEED-Bench ScienceQA-IMG VQAv2 MSVD-QA MSRVTT-QA ActivityNet-QA RefCOCO RefCOCO+
Metrics: accuracy compression retention rate CUDA time FLOPs KV cache storage
Baselines: Q-Former compression average pooling compression LLaMA-VID Video-LLaMA VideoChat Video-ChatGPT BT-Adapter Vista-LLaMA Chat-UniVi FrozenBiLM
Builds on: LLaVA LLaMA-VID Vicuna CLIP
Closest papers by meaning
- LLMC+: Benchmarking Vision-Language Model Compression with a plug-and-play Toolkit (AAAI 2026 · similarity 0.92)
- One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding (NeurIPS 2025 · similarity 0.91)
- Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.91)
- DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference (CVPR 2026 · similarity 0.90)
- Hybrid Token Compression for Vision-Language Models (CVPR 2026 · similarity 0.90)
- ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.