Research map / Vision-language and multimodal / Multimodal LLMs

RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inference

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RTPrune, a training-free two-stage visual token pruning method for DeepSeek-OCR that keeps high-norm tokens and merges the rest via Sinkhorn-based optimal transport, with a dynamic pruning ratio driven by inter-token similarity and Sobel-based textual density.

Problem

Visual tokens in DeepSeek-OCR contain redundancy, but existing VLM token pruning methods (attention-, similarity-, or text-relevance-based) fail to preserve OCR textual fidelity when applied to DeepSeek-OCR.

Approach

Observes that DeepSeek-OCR's LLM attends first to high-l2-norm visual tokens and then to the rest, so it keeps top-norm tokens and merges the remaining tokens into them via optimal transport (Sinkhorn with dustbin), while adapting the pruning ratio to feature similarity and edge-based textual density.

Key results

• Retains 99.47% of DeepSeek-OCR-Large performance on OmniDocBench with 84.25% token retention (15.75% pruned) and 1.23x faster prefill.
• On olmOCR-Bench with DeepSeek-OCR-Large at a fixed 300-token budget, RTPrune scores 68.3 vs 25.2 for FastV and 59.7 for VisionZip at 336 tokens.
• On DeepSeek-OCR-Base with 214 average tokens on OmniDocBench, RTPrune overall 81.48 vs 36.47 for DART and 8.12 for FastV (baseline 84.83).

Surprising findings

• Pruning existing VLM methods (FastV, SparseVLM) on DeepSeek-OCR performs worse than random pruning, producing repetitive or nonsensical output.
• Attention-based pruning inside the LLM is only viable when pruning in the final few layers; pruning in the first nine layers severely degrades OCR accuracy.

Limitations

• Dynamic pruning and fixed-ratio results show substantial degradation on some olmOCR-Bench subsets (e.g., multicolumn, long tiny text) at fixed 25% pruning.
• Evaluated mainly on DeepSeek-OCR variants and a few end-to-end OCR models; generalization beyond these is only partially demonstrated.

Experimental setup

Datasets: OmniDocBench olmOCR-Bench Ocean-OCR

Benchmarks: OmniDocBench olmOCR-Bench Ocean-OCR

Metrics: edit distance CDM TEDS accuracy GFLOPs prefill time

Baselines: FastV DART FitPrune DivPrune CDPruner SparseVLM VisionZip NÜWA

Builds on: DeepSeek-OCR VisionZip SuperGlue FastV

Closest papers by meaning

• When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs (CVPR 2026 · similarity 0.89)
• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.89)
• TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (CVPR 2026 · similarity 0.89)
• QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.89)
• Hi-Lo Prune: Look at What You'll Lose before Pruning with Hierarchical Token Selection (CVPR 2026 · similarity 0.88)
• TwinPrune: Density-Aware Two-Phase Token Pruning for Vision-Language Models (NeurIPS 2026 · similarity 0.88)