Research map / Vision-language and multimodal / Human and embodied modeling
RetFormer: Multimodal Retrieval for Enhancing Image Recognition
Tianrui Yu, Xiubo Liang, Hongzhi Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| retrieve both text and image instead of image only (ablation) | RetFormer · ImageNet-LT | Overall top-1 accuracy: 76.0 -> 81.9 | Improved |
| retrieve both text and image instead of text only (ablation) | RetFormer · ImageNet-LT | Overall top-1 accuracy: 78.1 -> 81.9 | Improved |
| add retrieval module to CLIP classifier fine-tuning | CLIP ViT-B/16 · ImageNet-LT | Few-shot top-1 accuracy: 52.9 (full FT) -> 76.8 | Improved |
| fine-tune the image encoder instead of freezing it (ablation) | RetFormer · ImageNet-LT | Overall top-1 accuracy: 78.6 -> 81.9 | Improved |
| increase cross-fusion layers from L=1 to L=4 (ablation) | RetFormer · ImageNet-LT | Overall top-1 accuracy: 81.2 -> 81.9 | Improved |
| increase cross-fusion layers from L=4 to L=8 (ablation) | RetFormer · ImageNet-LT | Overall top-1 accuracy: 81.9 -> 82.0 | No clear effect |
| increase retrieved neighbors k up to 32 (ablation) | RetFormer · ImageNet-LT | accuracy | Improved |
| increase k further (ablation) | RetFormer · ImageNet-LT | Many-shot accuracy | Hurt |
| enlarge the knowledge base (downstream, DataComp, All) (ablation) | RetFormer · ImageNet-LT | accuracy | Improved |
| use approximate HNSW index (recall 0.97) instead of exact k-NN (ablation) | RetFormer · ImageNet-LT | Overall top-1 accuracy: 81.9 -> 81.8 (23.34 ms -> 7.69 ms) | No clear effect |
| lower HNSW recall to 0.65 (ablation) | RetFormer · ImageNet-LT | Overall top-1 accuracy: 81.9 -> 78.7 | Hurt |
| use smaller k (8, 16) on Places-LT (ablation) | RetFormer · Places-LT | Many-shot accuracy: Many +0.3 to 0.8%, Few dipped | Mixed |
Closest papers by meaning
- Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrieval (CVPR 2025 · similarity 0.89)
- RMIR: A Benchmark Dataset for Reasoning-Intensive Multimodal Image Retrieval (CVPR 2026 · similarity 0.88)
- Your Embedding Model Is SMARTer Than You Think (NeurIPS 2026 · similarity 0.87)
- INQUIRE: A Natural World Text-to-Image Retrieval Benchmark (NeurIPS 2024 · similarity 0.87)
- LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.86)
- MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
