Research map / Vision-language and multimodal / Multimodal LLMs

Very Efficient Listwise Multimodal Reranking for Long Documents

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ZipRerank, an efficient listwise multimodal reranker that prunes query-relevant visual tokens and scores all candidates with one forward pass via single-logit decoding, trained with two-stage (text-rendered RankNet pretraining, then GPT-5-mini-distilled soft-ranking finetuning).

Problem

Listwise VLM-based reranking over long multimodal documents is accurate but slow, due to long visual-token prefills and autoregressive decoding of ranking/reasoning outputs.

Approach

Combines a non-parametric query-to-image max-cosine early-interaction token pruning with single-token logit scoring of candidate letters (one forward pass), and a two-stage training scheme with a geometric-decay soft-ranking cross-entropy loss distilled from a teacher VLM ranking.

Key results

• On MMDocIR with DSE top-20, ZipRerank reaches Recall@3 of 84.5 (micro) and Recall@5 of 89.4 at about 0.36 s cached LLM time, versus MM-R5's 79.0 and 83.9 at 3.82 s.
• MM-R5 keeps a slight edge on Recall@1 (64.1 vs 61.6 on the per-dataset Res. column for DSE), while ZipRerank leads on Recall@3/5 across most subsets.
• On ViDoRe English NDCG@5, ZipRerank improves over MM-R5 from 49.0 to 53.4 with DSE and from 55.8 to 59.9 with ColQwen.

Surprising findings

• Removing single-logit decoding (switching to autoregressive generation) leaves accuracy essentially unchanged while being over 6x slower.
• Pointwise LamRA achieves the highest ViDoRe NDCG@5 scores, above the listwise ZipRerank.

Limitations

• Query-aware pruning can discard fine-grained evidence such as small text, dense tables, or visually similar pages.
• Stage 2 relies on GPT-5-mini teacher rankings that may carry biases or errors.
• Evaluation focuses on document-image reranking benchmarks (MMDocIR, ViDoRe English), not other domains or languages.

Experimental setup

Datasets: RankZephyr MMDocIR training set MMDocIR ViDoRe

Benchmarks: MMDocIR ViDoRe

Metrics: Recall@1 Recall@3 Recall@5 NDCG@5 P@1 Mean Rank LLM time (s)

Baselines: MM-R5 LamRA UniME Llama-3.2-11B-Vision Qwen3-VL-8B-Instruct GPT-5-nano GPT-5-mini DocReRank UniME-V2

Builds on: RankNet MM-R5 Qwen3-VL DSE ColQwen

Closest papers by meaning

• LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.88)
• UniRank: Unified List-wise Reranking via Confidence-Ordered Denoising (NeurIPS 2026 · similarity 0.87)
• AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Reranking (NeurIPS 2025 · similarity 0.87)
• MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS (ICLR 2025 · similarity 0.87)
• Contrastive Retrieval Heads for Improved Attention-Based Reranking (NeurIPS 2026 · similarity 0.86)
• Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers (ICLR 2025 · similarity 0.86)