Research map / Vision-language and multimodal / Cross-modal learning
Closing the Indexing-Decoding Gap in Multimodal Generative Retrieval via Prefix Retention Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies an indexing-decoding gap in residual-quantization multimodal generative retrieval and proposes PRO (prefix ranking distillation, vocabulary scheduling, geometric score fusion) to improve target prefix survival under beam search.
Problem
Multimodal generative retrieval with residual-quantized identifiers and trie-constrained beam search irreversibly prunes target identifier prefixes, because identifier learning objectives do not enforce prefix discriminability during decoding.
Approach
Derives a survival bound linking prefix retention to ranking divergence, teacher margin and decoder mismatch, and uses a listwise prefix ranking distillation loss, ascending codebook sizes across RQ levels, and a geometric query-residual bias added to beam scores to improve prefix survival.
Key results
- PRO outperforms GENIUS on all nine M-BEIR-style tasks, e.g., COCO text-to-image R@1 45.8 vs 37.1 for GENIUS.
- On Flickr30k text-to-image, PRO reaches R@1 70.6 vs 56.8 for GENIUS and closes over 60% of the gap to dense retrieval.
- PRO improves all 12 Recall and NDCG metrics over TIGER on Amazon Beauty, Toys and Sports.
Surprising findings
- Identifiers with lower full-reconstruction MSE do not predict prefix survival; ECDFs of survived and pruned items nearly overlap.
- Ascending codebook sizes (small shallow levels, larger deep levels) outperform uniform and descending schedules, despite reducing early reconstruction capacity.
Limitations
- Analysis and method are specific to residual-quantization identifier pipelines.
- Assumes a two-stage setup with a fixed encoder and tokenizer, limiting joint optimization.
- Survival bound is conservative due to Pinsker-based analysis.
Experimental setup
Datasets: M-BEIR Flickr30k COCO WebQA NIGHTS CIRR OVEN InfoSeek Amazon Beauty Amazon Toys Amazon Sports
Benchmarks: M-BEIR Flickr30k COCO WebQA NIGHTS CIRR OVEN InfoSeek Amazon Beauty Amazon Toys Amazon Sports
Metrics: Recall@1 Recall@5 NDCG prefix survival rate
Baselines: CLIP-SF BLIP-FF IRGen GRACE AVG SemCORE GENIUS TIGER RIPOR
Builds on: GENIUS RQ-VAE TIGER trie-constrained beam search Pinsker's inequality
Closest papers by meaning
- GENIUS: A Generative Framework for Universal Multimodal Search (CVPR 2025 · similarity 0.88)
- MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction (ICLR 2026 · similarity 0.86)
- Your Embedding Model Is SMARTer Than You Think (NeurIPS 2026 · similarity 0.86)
- TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models (ICLR 2025 · similarity 0.86)
- BayesRAG: Probabilistic Mutual Evidence Corroboration for Multimodal Retrieval-Augmented Generation (NeurIPS 2026 · similarity 0.86)
- LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.