Research map / Label-efficient and robust learning / Recognition and long-tail
Evaluating Transferability in Retrieval Tasks: An Approach Using MMD and Kernel Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RetMMD, a training-free transferability metric for retrieval that uses kernel-based MMD between relevant and irrelevant document embeddings of pretrained models, averaged over queries, plus a retrieval-specific dataset distance.
Problem
Selecting the best pretrained model or source dataset for retrieval tasks normally requires costly fine-tuning; existing transfer learning evaluation metrics target classification and do not directly apply to query-dependent retrieval.
Approach
Treat fine-tuning as learning a kernel mapping, and measure MMD between per-query relevant and irrelevant document embeddings on pretrained embeddings (with kernel selection via two-sample tests or SVM), averaged with inverse-frequency query weights, to predict post-fine-tuning retrieval performance; also measure dataset distance via per-query mean embeddings.
Key results
- RetMMD-M achieves weighted Kendall correlations of 0.90 on CUB200, 0.89 on CARS196, 0.80 on SOP, 0.94 on NFCorpus, and 1.00 on COCO, outperforming LogME and GBC on most datasets.
- On NFCorpus, RetMMD with RBF kernel is not sensitive to zero-shot performance; OpenAI text-embedding-ada-002 gets a much larger RetMMD score (316.86 with linear kernel) consistent with its strong zero-shot retrieval.
- RetMMD evaluation on SOP with five pretrained image models on one A10 GPU is much faster than directly fine-tuning those models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RetMMD-M rather than the compared transferability metrics | Retrieval transferability evaluation · CUB200 | weighted Kendall rank correlation: 0.77 ± 0.10 -> 0.90 ± 0.08 | Helped |
| Use RetMMD-M rather than the compared transferability metrics | Retrieval transferability evaluation · CARS196 | weighted Kendall rank correlation: 0.75 ± 0.13 -> 0.89 ± 0.08 | Helped |
| Use RetMMD-M rather than the compared transferability metrics | Retrieval transferability evaluation · SOP | weighted Kendall rank correlation: 0.71 ± 0.10 -> 0.80 ± 0.13 | Helped |
| Use RetMMD-M rather than the compared transferability metrics | Retrieval transferability evaluation · NFCorpus | weighted Kendall rank correlation: 0.70 ± 0.13 -> 0.94 ± 0.00 | Helped |
| Use RetMMD-M rather than GBC | Retrieval transferability evaluation · FEVER | weighted Kendall rank correlation: 0.67 ± 0.09 -> 0.52 ± 0.05 | Hurt |
| Use RetMMD-M rather than the compared transferability metrics | Retrieval transferability evaluation · FiQA-2018 | weighted Kendall rank correlation: 0.33 ± 0.00 -> 0.80 ± 0.02 | Helped |
| Use RetMMD-M rather than the compared transferability metrics | Retrieval transferability evaluation · COCO | weighted Kendall rank correlation: −0.09 ± 0.14 -> 1 ± 0.00 | Helped |
| Use a polynomial kernel with lower-degree moment matching | RetMMD · CARS196 | weighted Kendall rank correlation | Helped |
| Use a polynomial kernel with higher-degree moment matching | RetMMD · COCO | weighted Kendall rank correlation | Helped |
| Calculate dataset distance using mean embeddings of relevant documents per query | Dataset transferability evaluation · Image datasets transferred from ImageNet | Pearson correlation | Helped |
Surprising findings
- With a linear kernel, RetMMD tracks zero-shot performance rather than post-fine-tuning performance, so kernel choice materially changes the metric's behavior.
- On CUB200 the RetMMD-S (single-kernel) score correlates with fine-tuning performance, yet the paper notes that on the text datasets nearly all kernels yield significant p-values, so kernel selection is not discriminative there.
Limitations
- Relevant documents are treated as equally weighted; ranking or document importance is not modeled.
- Kernel selection is heuristic (two-sample test, SVM, curvature), and the choice materially changes results; some text settings yield near-100% significant p-values for all kernels.
- Retrieval is assumed to depend only on embedding similarity; the metric does not cover retrieval that goes beyond embedding similarity.
Experimental setup
Datasets: CUB200 CARS196 Stanford Online Products NFCorpus FiQA-2018 FEVER MS-COCO ImageNet-1K Stanford Dogs SVHN CIFAR-10 CIFAR-100
Benchmarks: CUB200 CARS196 SOP NFCorpus FEVER FiQA-2018 COCO
Metrics: weighted Kendall rank correlation Pearson correlation Recall@1 MAP@1 F1 score
Baselines: LogME GBC
Builds on: LogME GBC MMD LEEP
Closest papers by meaning
- MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS (ICLR 2025 · similarity 0.85)
- NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.84)
- A Statistical Framework for Data-dependent Retrieval-Augmented Models (ICML 2024 · similarity 0.84)
- Combining Multiple Supervision for Robust Zero-Shot Dense Retrieval (AAAI 2024 · similarity 0.84)
- LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.84)
- Hierarchical Retrieval: The Geometry and a Pretrain-Finetune Recipe (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.