Research map / Label-efficient and robust learning / Recognition and long-tail
Hierarchical Retrieval: The Geometry and a Pretrain-Finetune Recipe
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that dual encoders can represent hierarchical retrieval (ancestor retrieval on DAGs) with logarithmic dimension, and proposes a pretrain-finetune recipe (finetune on long-distance pairs) that fixes the lost-in-the-long-distance failure on WordNet and ESCI.
Problem
Dual encoders for hierarchical retrieval, where a query must retrieve all its ancestors in a hidden DAG, are limited by Euclidean geometry and low embedding dimension; long-distance ancestors are often lost in retrieval.
Approach
Constructive random-Gaussian embeddings prove that asymmetric DEs with dimension O(s log m) solve HR, and a pretrain-finetune recipe (pretrain on regular pairs, then finetune on long-distance pairs only with lower learning rate and higher temperature) improves long-distance recall without rebalancing hurting short-distance recall.
Key results
- On WordNet with 64-dim lookup-table DE, pretrain-finetune reaches 92.3% overall recall versus 71.4% for regular sampling and 41.8% for rebalanced (p=0.01).
- On the synthetic H=4, W=5 tree with d=3, pretrain-finetune raises overall recall to 97% versus 66% for regular and 70% for rebalanced training.
- On HyperLex, pretrain-finetune gets Spearman 0.415 versus 0.350 for regular sampling and 0.389 for WN-Euclidean.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain-finetune recipe (pretrain on regular data, finetune on long-distance heavy-tail data) instead of regular sampling | lookup-table DE on WordNet · WordNet, dim 16/32/64 | overall recall: +17.1 (16d), +25.5 (32d), +20.9 (64d) | Helped |
| rebalancing training data with heavy-tail mixing (p=0.01 or 0.03) (ablation) | lookup-table DE on WordNet · WordNet, dim 64 | overall recall and distance-0 recall: overall 71.4 -> 41.8 (p=0.01) and 40.7 (p=0.03); distance-0 recall 100.0 -> 0.6 / 2.8 | Hurt |
| pretrain-finetune recipe instead of regular sampling | DE on WordNet hypernymy (HyperLex) · HyperLex Spearman rho, dim 5 | Spearman rho: 0.350 -> 0.415 | Helped |
| pretrain-finetune recipe with long-distance finetuning on synthetic tree (H=4, W=5, d=3) (ablation) | lookup-table DE on synthetic tree · synthetic tree overall recall | overall recall across distances: 66% -> 97% | Helped |
| hyperbolic embeddings instead of Euclidean (dim 16, standard training) | lookup-table DE on WordNet · WordNet, dim 16 | overall recall: 43.0 -> 45.4 (still below pretrain-finetune 60.1) | Helped |
| joint training on exact and substitute ESCI matches vs pretrain on Exact then finetune on Substitute | DE on ESCI · ESCI Exact and Substitute recall@k | recall@k for k in {100,1000,10000}: qualitative; pretrain-finetune better than joint training | Helped |
| finetuning learning rate multiplier increased from 1e-7 to 1e-3 (ablation) | pretrain-finetune on WordNet · WordNet long-distance and short-distance recall | recall | Mixed |
Surprising findings
- Rebalancing training data toward long-distance pairs fails: it improves distance 1-2 recall but substantially degrades distance-0 recall, and tuning the mixing ratio p does not resolve the tradeoff.
- Hyperbolic embeddings (16-dim) on WordNet reach only 45.4% overall recall, better than Euclidean standard training (43.0%) but well below the pretrain-finetune recipe (60.1%).
Limitations
- Theoretical construction requires the DAG as input; learned embeddings only empirically validated, not proven.
- Finetuning on long-distance data only degrades distance-0 recall over time, requiring early stopping on a validation set.
- Recipe assumes sufficient short-distance data for meaningful pretraining; ESCI setting only loosely matches the HR definition.
Experimental setup
Datasets: WordNet noun hierarchy (82,115 synsets) ESCI shopping queries HyperLex
Benchmarks: WordNet hypernym retrieval ESCI HyperLex
Metrics: recall@k recall by query-document distance Spearman correlation
Baselines: Regular sampling Rebalanced sampling OrderEmb WN-Basic WN-Euclidean Hyperbolic embeddings
Builds on: dual encoder softmax loss for retrieval Lazy Adam
Closest papers by meaning
- USTAD: Unified Single-model Training Achieving Diverse Scores for Information Retrieval (ICML 2024 · similarity 0.87)
- Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT (ICML 2024 · similarity 0.87)
- Adaptive Retrieval and Scalable Indexing for k-NN Search with Cross-Encoders (ICLR 2024 · similarity 0.87)
- MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encoding (NeurIPS 2024 · similarity 0.86)
- Fine-Grained Distillation for Long Document Retrieval (AAAI 2024 · similarity 0.86)
- NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.