Research map / Vision-language and multimodal / Human and embodied modeling
Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReT, a Transformer-based recurrent cell with LSTM-style sigmoid gates that fuses multi-layer visual and textual CLIP features on both query and document sides for multimodal (image+text) document retrieval with fine-grained late interaction.
Problem
Retrieving documents for multimodal queries (image plus text) from collections where text and images are interleaved; existing feature-fusion retrievers use only last-layer features and lose low-level visual detail.
Approach
A recurrent cell that iterates over layers of frozen vision and text backbones, using LSTM-inspired forget and modality-specific input gates to merge multi-level features into a set of k latent tokens, scored via fine-grained max-sim late interaction and trained with a symmetric InfoNCE variant on separate query and document encoders.
Key results
- On M2KR with CLIP ViT-L, ReT reaches 22.2 R@5 points above PreFLMR on OVEN (82.0 vs 59.8) and outperforms baselines on most datasets.
- On M-BEIR InfoSeek and OVEN with multimodal queries, ReT trained only on M2KR reaches R@5 28.3 on InfoSeek vs UniIR's 22.4 (document-text only setting).
- Removing multi-layer recurrence (single final-layer application) drops M2KR averages substantially, e.g. OVEN R@5 from 82.0 to 80.6 and LLaVA R@1 from 79.9 to 74.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ReT instead of CLIP (Feature Fusion) | multimodal retrieval with CLIP ViT-L · M2KR OVEN | R@5: 76.0 -> 82.0 | Helped |
| use ReT instead of CLIP (Feature Fusion) | multimodal retrieval with CLIP ViT-L · M2KR WIT | R@10: 68.2 -> 73.4 | Helped |
| use ReT instead of CLIP (Feature Fusion) | multimodal retrieval with CLIP ViT-L · M2KR IGLUE | R@1: 76.9 -> 81.8 | Helped |
| use ReT instead of CLIP (Feature Fusion) | multimodal retrieval with CLIP ViT-L · M2KR KVQA | R@5: 47.5 -> 63.5 | Helped |
| use ReT instead of CLIP (Feature Fusion) | multimodal retrieval with CLIP ViT-L · M2KR InfoSeek | PR@5: 54.7 -> 60.5 | Helped |
| use ReT instead of CLIP (Feature Fusion) | multimodal retrieval with CLIP ViT-L · M2KR OKVQA | PR@5: 59.4 -> 66.2 | Helped |
| fine-tune on M-BEIR | ReT · M-BEIR OVEN, multimodal documents | R@5: 51.9 -> 56.3 | Helped |
| fine-tune on M-BEIR | ReT · M-BEIR InfoSeek, multimodal documents | R@5: 30.8 -> 36.5 | Helped |
| replace fine-grained loss with standard contrastive loss (ablation) | ReT · M2KR OVEN | R@5: 82.0 -> 80.6 | Hurt |
| add candidate document images (ablation) | ReT · M2KR OVEN | R@5: 77.5 -> 82.0 | Helped |
| add multi-layer recurrence (ablation) | ReT · M2KR OVEN | R@5: 80.6 -> 82.0 | Helped |
| apply recurrence only in the first 4 layers instead of all layers (ablation) | ReT · M2KR OVEN | R@5: 82.0 -> 54.2 | Hurt |
Surprising findings
- Applying the recurrent cell only to the first four backbone layers performs far worse than applying it to the last four, contrary to the intuition that shallow layers provide useful low-level detail.
- Larger OpenCLIP ViT-H backbone performs worse than CLIP ViT-L on several datasets, suggesting larger visual encoders do not universally help.
Limitations
- Document images for OVEN, InfoSeek, E-VQA and OKVQA were added by the authors, and the enriched benchmark is not described as released.
- Lower PR@5 than PreFLMR on E-VQA and OKVQA, attributed to competitor batch sampling strategies.
- Only the ReT approach is evaluated on M-BEIR splits with multimodal queries; other M-BEIR tasks are not covered.
Experimental setup
Datasets: M2KR MSMARCO WIT IGLUE KVQA CC3M OVEN LLaVA OKVQA InfoSeek Encyclopedic-VQA M-BEIR
Benchmarks: M2KR M-BEIR InfoSeek (RAG)
Metrics: R@K PR@K (pseudo recall) R@1 R@5 R@10 inference time
Baselines: CLIP (Feature Averaging) CLIP (Unimodal) CLIP (Feature Fusion) FLMR PreFLMR UniIR SigLIP BLIP-2
Builds on: CLIP ColBERT UniIR PreFLMR LSTM Block-Recurrent Transformer
Closest papers by meaning
- LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.89)
- Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.88)
- Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (CVPR 2025 · similarity 0.88)
- Your Embedding Model Is SMARTer Than You Think (NeurIPS 2026 · similarity 0.87)
- Generalized Contrastive Learning for Universal Multimodal Retrieval (NeurIPS 2025 · similarity 0.87)
- Improving Context Understanding in Multimodal Large Language Models via Multimodal Composition Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.