Research map / Vision-language and multimodal / Multimodal LLMs
Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Prune2Drive prunes visual tokens for multi-view driving VLMs using a farthest-point-sampling token selector (T-FPS) plus a TPE-optimized per-view pruning ratio controller, training-free and compatible with FlashAttention.
Problem
Multi-view, high-resolution driving VLMs produce many visual tokens, causing high prefilling latency, memory use and quadratic attention cost; existing token pruning methods are single-image oriented and rely on attention scores that are incompatible with efficient attention kernels.
Approach
A diversity-aware token selector that adapts farthest point sampling with cosine distance in embedding space, combined with a view-adaptive controller that searches per-camera retention ratios with a reward-penalty objective on a small validation subset, requiring no retraining or attention maps.
Key results
- With 10% of visual tokens retained on DriveMM, Prune2Drive reaches 6.40x prefill speedup and 13.4% FLOPs, with average score 57.4 vs. 59.1 vanilla on DriveLM.
- On DriveLMM-o1 with 25 tokens per image (90% pruning), Prune2Drive achieves overall reasoning 68.3 vs. 74.2 vanilla and 67.4 for the next best (DART).
- On OmniDrive (10% tokens), Prune2Drive averages 49.0 vs. 50.3 vanilla, outperforming FastV (44.3) and SparseVLM (46.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Prune2Drive token selection (T-FPS + view-adaptive ratios) instead of baselines at 75% pruning | DriveLMM-o1 · DriveLMM-o1 benchmark, 64 tokens/image | Overall Reasoning: 69.3 vs 68.6 (FastV), 68.4 (DART), 68.3 (PACT) | Helped |
| Prune2Drive token selection instead of baselines at 90% pruning | DriveLMM-o1 · DriveLMM-o1 benchmark, 25 tokens/image | Overall Reasoning: 68.3 vs 67.4 (DART), 67.0 (PACT), 65.3 (FastV) | Helped |
| Replace cosine distance with nearest-token selection (min distance) in T-FPS (ablation) | T-FPS token selection · DriveLMM-o1 benchmark, 90% pruned | Overall Reasoning: 63.0 vs 68.3 with cosine | Hurt |
| Replace cosine distance with nearest-token selection (min distance) in T-FPS (ablation) | T-FPS token selection · DriveLM, 90% pruned | Average: 52.3 vs 57.4 with cosine | Hurt |
| Replace cosine distance with l1 or l2 distance in T-FPS (ablation) | T-FPS token selection · DriveLM and DriveLMM-o1, 90% pruned | Average / Overall Reasoning: DriveLM 56.9 (l1), 57.1 (l2) vs 57.4; DriveLMM-o1 68.3 (l1), 67.7 (l2) vs 68.3 | No clear effect |
| TPE instead of grid search or evolutionary search for pruning ratio optimization (ablation) | View-adaptive pruning ratio optimization · DriveLM and DriveLMM-o1, 90% pruned | Average / Overall Reasoning: DriveLM 57.4 vs 57.2 (grid), 57.1 (evo); DriveLMM-o1 68.3 vs 67.3 (grid), 67.6 (evo) | Helped |
| Prune2Drive vs FastV on multi-view DriveMM efficiency (10% tokens) | DriveMM · DriveLM efficiency, 438 tokens | Prefill speedup: 6.40x vs 5.78x (FastV), 4.06x (SparseVLM) | Helped |
| Prune2Drive on OmniDrive video benchmark at 10% visual tokens | DriveMM · OmniDrive video AD | Average: 49.0 vs 46.8 (SparseVLM), 44.3 (FastV); vanilla 50.3 | Helped |
Surprising findings
- Nearest-token selection (inverse of the proposed rule) causes large performance degradation, while the choice between l1, l2 and cosine distances has minimal effect.
- Prune2Drive reports slightly higher match score than the unpruned model on DriveLM at 25% retention (34.0 vs 33.9).
Limitations
- Evaluation limited to offline open-loop AD datasets; no closed-loop evaluation.
- T-FPS may under-sample safety-critical objects covering large uniform-color regions (failure cases shown).
- Pruning ratio search depends on a validation subset and a GPT-based or similarity-based reward; the theoretical bound relies on an assumed Lipschitz-style model with unverified view weights.
Experimental setup
Datasets: DriveLM DriveLMM-o1 OmniDrive GQA VQA V2 POPE VizWiz MME NuScenes
Benchmarks: DriveLM DriveLMM-o1 OmniDrive GQA VQA V2 POPE VizWiz MME
Metrics: accuracy BLEU-4 ROUGE-L CIDEr ChatGPT score match score risk assessment accuracy overall reasoning prefill speedup decode speedup
Baselines: FastV DART SparseVLM PACT
Builds on: FastV DART SparseVLM PACT Farthest Point Sampling
Closest papers by meaning
- Each Complexity Deserves a Pruning Policy (NeurIPS 2025 · similarity 0.93)
- EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models (NeurIPS 2025 · similarity 0.90)
- Less Language, More Latents: Annotation-Efficient VLAs for Driving (NeurIPS 2026 · similarity 0.90)
- LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models (ICLR 2026 · similarity 0.90)
- TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.90)
- QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.