Research map / Vision-language and multimodal / Multimodal LLMs
FastVLM: Efficient Vision Encoding for Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FastVLM, a VLM built on a new hybrid convolutional-transformer encoder FastViTHD that emits few visual tokens at high resolution, cutting time-to-first-token while matching or beating prior VLMs.
Problem
High-resolution visual encoding in VLMs is slow and produces many visual tokens, inflating LLM prefilling and time-to-first-token (TTFT), so accuracy-latency trade-offs are poor for text-rich tasks.
Approach
Use a hybrid conv-transformer encoder (FastViTHD) with an extra downsampling stage so self-attention runs at 1/32 resolution, producing 4x fewer tokens than FastViT and 16x fewer than ViT-L/14 at 336; scale input resolution directly rather than using token pruning, and measure latency on-device to map the resolution-LLM-size Pareto frontier.
Key results
- In the LLaVA-1.5 setup, FastVLM achieves 3.2x TTFT improvement while maintaining similar benchmark performance to prior works.
- FastVLM (0.5B LLM, 1024 res) outperforms LLaVA-OneVision at 1152 res on SeedBench, MMMU, and DocVQA with 85x faster TTFT (166 ms vs 14124 ms) and a 3.4x smaller encoder.
- FastViTHD (125M params) reaches 6.8 ms encoder latency at 256 res with 16 tokens for Vicuna-7B, versus 127.4 ms for ViT-L/14 at 336 res with 576 tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ViT-L/14 with FastViT at 768×768 | LLaVA-1.5 with Vicuna-7B · DocVQA | score: 28.7 -> 34.4 | Helped |
| Add multi-scale features with AvgPool (ablation) | FastViT VLM · Avg-5 | score: 62.6 -> 62.7 | Helped |
| Add multi-scale features with depthwise convolutions (ablation) | FastViT VLM · Avg-5 | score: 62.6 -> 62.9 | Helped |
| Use multi-scale depthwise convolutions instead of AvgPool (ablation) | FastViT VLM · Avg-5 | score: 62.7 -> 62.9 | Helped |
| Replace FastViT with FastViTHD | VLM across resolution and LLM-size pairs · Average-5 | score: over 2.5 points | Helped |
| Switch the decoder from Vicuna-7B to Qwen2 | FastVLM · VLM benchmarks | score | Helped |
| Increase image resolution from 256 to 1024 | FastVLM · GQA | accuracy | Helped |
| Add a 15M-sample resolution-scaling pretraining stage | FastVLM · VLM benchmarks | score | Helped |
| Increase instruction-tuning data from 1.1M to 12.5M | FastVLM · VLM benchmarks | score | Helped |
| Use static high-resolution input instead of tiled dynamic resolution | FastViTHD VLM · Accuracy-latency trade-off | trade-off | Helped |
| Scale input resolution using dynamic resolution with more tiles (ablation) | FastVLM · VLM performance | score | Hurt |
| Include squeeze-excite layers in patch embedding blocks (ablation) | FastViTHD · High-resolution inference | inference latency | Hurt |
Surprising findings
- Even with 16 visual tokens (256 input resolution), FastViTHD outperforms recent token-pruning methods on ViT-L/14 that retain 40-192 tokens.
- Pairing high resolution with a small LLM is suboptimal; the Pareto frontier consists of varying LLM sizes, and dynamic AnyRes tiling is only beneficial at very high resolutions with few tiles.
Limitations
- Training data for the best models relies on recaptioned and large instruction-tuning mixtures, and some results use different data scales, complicating fair comparison.
- Latency is benchmarked only on a specific Apple M1 Max setup (Core ML encoder, MLX LLM).
Experimental setup
Datasets: DataCompDR-1B LLaVA-1.5 558K LLaVA-1.5 665K Recap-CC3M Recap-CC12M Cambrian-7M LLaVA-OneVision data DocMatix MammothVL
Benchmarks: GQA ScienceQA TextVQA POPE LLaVA-in-the-wild VQAv2 MMVet MMMU DocVQA SeedBench ChartQA OCRBench
Metrics: accuracy TTFT latency (ms) vision encoder latency (ms) Avg-5
Baselines: LLaVA-OneVision LLaVA-1.5 ConvLLaVA Cambrian-1 MM1 LLaVA-NeXT SmolVLM2 FlorenceVL VisionZip SparseVLM FastV DynamicLLaVA
Builds on: LLaVA-1.5 FastViT MobileCLIP RepMixer Qwen2
Closest papers by meaning
- iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.91)
- Dense Connector for MLLMs (NeurIPS 2024 · similarity 0.90)
- DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference (CVPR 2026 · similarity 0.89)
- A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.89)
- Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance (CVPR 2026 · similarity 0.89)
- VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.