Research map / Vision-language and multimodal / Human and embodied modeling
QuARI: Query Adaptive Retrieval Improvement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QuARI, a transformer hypernetwork that predicts a query-specific low-rank linear transformation of frozen VLM image embeddings for fast instance and text-to-image retrieval and re-ranking.
Problem
Vision-language models with global embeddings perform poorly on challenging retrieval such as instance retrieval at very large scale and expert-level text-to-image search, while existing re-ranking methods are computationally expensive.
Approach
A transformer-based hypernetwork conditioned on each query embedding predicts a rank-64 linear transformation (via iteratively refined U/V tokens) that is applied to the precomputed gallery embeddings, plus a refined query embedding; trained with a contrastive loss using semi-positive samples mined from nearest neighbors and query noise to bridge the modality gap.
Key results
- On ILIAS, SigLIP2 ViT-L with QuARI reaches 35.3 I2I mAP@1k at 100M (vs 20.8 baseline) and 40.6 T2I at 5M (vs 24.7).
- On INQUIRE text-to-image re-ranking, SigLIP2 + QuARI gets 45.6 mAP@50, above the best open-source VLM (VILA-40B, 40.2) and GPT-4o (43.7).
- On ILIAS top-1k image re-ranking, QuARI achieves 29.1 mAP@1k vs 26.4 for AMES and 19.6 initial ranking.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add query-specific QuARI adaptation to SigLIP2 ViT-L | SigLIP2 ViT-L · ILIAS I2I @ 100M | mAP@1k: 20.8 -> 35.3 (+14.5) | Helped |
| add query-specific QuARI adaptation to SigLIP2 ViT-L | SigLIP2 ViT-L · ILIAS T2I @ 5M | mAP@1k: 24.7 -> 40.6 (+15.9) | Helped |
| add query-specific QuARI adaptation to SigLIP2 ViT-L | SigLIP2 ViT-L · ILIAS T2I @ 100M | mAP@1k: 19.8 -> 38.2 (+18.4) | Helped |
| add query-specific QuARI adaptation | SigLIP SoViT-400m · INQUIRE | mAP@50: 34.2 -> 45.4 (+11.2) | Helped |
| use one-step generation instead of iterative generation (ablation) | QuARI · ILIAS I2I @ 100M | mAP@1k: 35.3 -> 28.5; 6.8 decrease | Hurt |
| use one-step generation instead of iterative generation (ablation) | QuARI · ILIAS T2I @ 5M | mAP@1k: 40.6 -> 32.1; 8.5 decrease | Hurt |
| use one-step generation instead of iterative generation (ablation) | QuARI · INQUIRE | mAP@50: 50.7 -> 43.8; 6.9 decrease | Hurt |
| remove semi-positive samples during training (ablation) | QuARI · ILIAS I2I @ 100M | mAP@1k: 35.3 -> 30.3; 5.0 decrease | Hurt |
| remove semi-positive samples during training (ablation) | QuARI · INQUIRE | mAP@50: 50.7 -> 45.9; 4.8 decrease | Hurt |
| remove query noise during training (ablation) | QuARI · ILIAS I2I @ 100M | mAP@1k: 35.3 -> 20.0; 15.3 decrease | Hurt |
| remove query noise during training (ablation) | QuARI · ILIAS T2I @ 5M | mAP@1k: 40.6 -> 32.4; 8.2 decrease | Hurt |
| remove query noise during training (ablation) | QuARI · INQUIRE | mAP@50: 50.7 -> 44.1; 6.6 decrease | Hurt |
Surprising findings
- Without query noise during training, QuARI image-to-image performance on ILIAS 100M (20.0) falls below the SigLIP2 baseline (20.8).
- Fine-tuning SigLIP2 on the same training data gives only a small gain (20.8 to 21.1 I2I on ILIAS).
Limitations
- Linear transformation restricts expressiveness; cannot improve if the original feature space lacks relevant features.
- Re-ranking is bounded by the top-k initial retrieval set and cannot recover items outside it.
Experimental setup
Datasets: MS COCO Conceptual Captions 12M BioTrove Universal Embeddings ILIAS INQUIRE YFCC100M
Benchmarks: ILIAS INQUIRE
Metrics: mAP@1k mAP@50 nDCG@50 MRR
Baselines: OAI CLIP SigLIP SigLIP2 OpenCLIP static task adaptation alphaQE Chamfer Similarity Spatial Verification AMES LLaVA-1.5-13B LLaVA-v1.6-7B LLaVA-v1.6-34B
Builds on: HyperCLIP LinCIR HyperDreamBooth Transformers as Meta-Learners CLIP
Closest papers by meaning
- Matryoshka Query Transformer for Large Vision-Language Models (NeurIPS 2024 · similarity 0.86)
- Exploiting Distribution Constraints for Scalable and Efficient Image Retrieval (ICLR 2025 · similarity 0.86)
- LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.86)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.86)
- QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models (NeurIPS 2025 · similarity 0.86)
- Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.