Research map / Vision-language and multimodal / Multimodal LLMs
FlashSloth : Lightning Multimodal Large Language Models via Embedded Visual Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlashSloth, a tiny MLLM (phi2-2.7B LLM, SigLIP encoder) that uses spatial attention pooling and an embedded query module to cut visual tokens to 81-90 while keeping competitive multimodal performance and much faster response time.
Problem
Tiny MLLMs still use many visual tokens, which limits their real inference speedup and increases training cost and latency.
Approach
Compress visual tokens with a spatial attention pooling module that keeps visually salient content, plus a lightweight embedded query module (EmbQ) inserted inside the LLM that pulls instruction-relevant visual details via cross-attention, avoiding a separate Q-Former-style bridge and dedicated alignment pretraining.
Key results
- FlashSloth uses 90 visual tokens on GQA versus 414 for InternVL2 and 1699 for Qwen2-VL-2B, cutting response time to about 0.05s, roughly 2-5x faster than baselines.
- FlashSloth-HD reaches 75.7 on MMB and 49.0 on MM-Vet, matching or exceeding InternVL2-2B on several benchmarks while using less compute.
- Ablation: average pooling to 81 tokens drops MMB from 67.9 to 62.7, while SAP plus EmbQ (90 tokens) recovers MMB to 67.9 with only 0.09 TFLOPs versus 0.30 for the 729-token baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace visual tokens with spatial attention pooling plus EmbQ (81 visual tokens + 9 queries) versus uncompressed 729 tokens (ablation) | FlashSloth · MMBench | accuracy: 67.9 vs 67.9 | No clear effect |
| Replace SAP with simple average pooling (81 tokens) (ablation) | FlashSloth · MMBench | accuracy: -5.2 vs uncompressed (62.7 vs 67.9) | Hurt |
| Add spatial attention pooling (SAP) instead of average pooling (ablation) | FlashSloth · MMBench | accuracy: +1.2 (63.9 vs 62.7) | Helped |
| Add EmbQ on top of SAP (ablation) | FlashSloth · MMBench | accuracy: +4.0 (67.9 vs 63.9) | Helped |
| Add EmbQ on top of SAP (ablation) | FlashSloth · GQA | accuracy: +0.9 (60.8 vs 60.2) | Helped |
| Initialize EmbQ queries from dot token embedding instead of random init (ablation) | FlashSloth · MMBench | accuracy: +2.3 (67.9 vs 65.6) | Helped |
| Use 9 EmbQ queries instead of 0 queries (ablation) | FlashSloth · MMBench | accuracy: +4.0 (67.9 vs 63.9) | Helped |
| Increase EmbQ queries beyond 9 (12 or 15) (ablation) | FlashSloth · MMBench | accuracy: 67.9 at 9 vs 67.8 at 12 and 67.0 at 15 | No clear effect |
| FlashSloth (SAP+EmbQ, 90 tokens) vs LLaVA-1.5-7B and tiny MLLMs in efficiency; reduces visual tokens by 80-89% vs tiny MLLMs | FlashSloth · GQA response time and throughput (Table 1) | response time (s): 0.05 vs 0.11 (LLaVA-1.5-7B) and 0.24 (InternVL2) | Helped |
| Use EmbQ-augmented FlashSloth with SAP instead of full visual tokens on MMBench performance vs other tiny MLLMs | FlashSloth · MMBench | accuracy: 73.0 vs InternVL2 73.2 and Qwen2-VL 74.9; FlashSloth-HD 75.7 | Mixed |
| Use high-resolution input (FlashSloth-HD, 768px, 405 tokens) instead of 384px default | FlashSloth · DocVQA | accuracy: +18.8 (69.8 vs 51.0) | Helped |
| Remove EmbQ/SAP compression and use 729 tokens vs SAP-only 81 tokens for GQA (ablation) | FlashSloth · GQA | accuracy: -1.0 vs uncompressed (60.8 vs 61.8) | Hurt |
Surprising findings
- Tiny MLLMs with many visual tokens can have slower first-token response than LLaVA-1.5-7B despite fewer parameters.
- EmbQ needs only 9 learnable queries to recover baseline performance, and random query initialization works nearly as well as dot-token initialization.
Limitations
- Lags behind Qwen2-VL and InternVL2 on OCR-heavy benchmarks such as DocVQA at default resolution
- FlashSloth-HD requires high-resolution inputs and an unfrozen vision encoder, raising training cost
- Token pruning methods are treated as orthogonal and not combined with the proposed design
Experimental setup
Datasets: LLaVA-665k LLaVA pretraining split (1.2M)
Benchmarks: MMB MME MM-Vet POPE SEED MMMU MathVista GQA SQA TextVQA AI2D ChartQA
Metrics: accuracy response time (first token) TFLOPs GPU memory throughput
Baselines: Qwen2-VL-2B InternVL2-2B MiniCPM-V2 MobileVLM-V2 IMP-v1 MM1.5 LLaVA-1.5-7B Mini-Gemini
Builds on: LLaVA-1.5 Q-Former LLaVA-NeXT
Closest papers by meaning
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.91)
- LLMC+: Benchmarking Vision-Language Model Compression with a plug-and-play Toolkit (AAAI 2026 · similarity 0.89)
- Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.88)
- ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming (AAAI 2025 · similarity 0.88)
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.88)
- LensVLM: Selective Context Expansion for Compressed Visual Representation of Text (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.