Research map / Label-efficient and robust learning / Recognition and long-tail

Exploiting Distribution Constraints for Scalable and Efficient Image Retrieval

ICLR 2025 · Label-efficient and robust learning › Recognition and long-tail · approach: Correspondence and causal representation learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AE-SVC, an unsupervised autoencoder with covariance, variance and mean constraints on the latent space to improve off-the-shelf foundation-model embeddings for image retrieval, and (SS)2D, a single-shot similarity-space distillation that yields adaptive nested embeddings.

Problem

Off-the-shelf foundation model embeddings underperform dataset-specific retrieval models, and large embeddings make cosine-similarity retrieval slow; existing dimensionality reduction (PCA, AE, VAE) is not retrieval-oriented and needs separate training per size.

Approach

Constrain an autoencoder's latent space to be orthogonal, zero-mean, and unit-variance so the cosine similarity distribution has lower variance and is more discriminative; then distill the full latent's pairwise cosine-similarity structure into nested prefix embeddings of many sizes trained jointly (single-shot).

Key results

• AE-SVC gives an average 15.5% improvement over PCA at full embedding size across datasets, with up to 24% on InShop for DINOv2-Large.
• (SS)2D yields up to a further 10% improvement over AE-SVC at smaller embedding sizes, approaching the per-dimension SSD upper bound.
• AE-SVC with 3-layer encoder-decoder (AE-SVC3) reaches 0.568 map@4 at size 384 on InShop with DINO-S8, versus 0.411 for PCA.

Measured effects reported in the paper

Surprising findings

• AE-SVC applied to a dataset-specific model (CosPlace) gives only about 2% gain, versus about 10% for off-the-shelf Dino, so the benefit is concentrated in foundation models.
• Deeper AE-SVC networks (10 layers) degrade sharply, and applying constraints at later decoder layers lowers performance.

Limitations

• AE-SVC gives limited gains for dataset-specific models that already capture retrieval-relevant structure.
• (SS)2D adds training overhead from computing losses for many embedding sizes.
• Theoretical analysis relies on Gaussian and large-dimension approximations that are only checked empirically.

Experimental setup

Datasets: InShop Stanford Online Products Pittsburgh30K TokyoVal LAION-2B ImageNet LVD-142M

Benchmarks: InShop Stanford Online Products Pittsburgh30K TokyoVal

Metrics: mAP@k Recall@k

Baselines: PCA PCA Whitening TLDR Variational Autoencoder SSD CosPlace

Builds on: Barlow Twins VICReg TLDR PCA Whitening Similarity-Preserving Knowledge Distillation

Closest papers by meaning

• Thinking Fast and Slow: Efficient Text-to-Visual Retrieval With Transformers (CVPR 2021 · similarity 0.88)
• On the Theoretical Limitations of Embedding-Based Retrieval (ICLR 2026 · similarity 0.88)
• NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.87)
• QuARI: Query Adaptive Retrieval Improvement (NeurIPS 2025 · similarity 0.86)
• Your Embedding Model Is SMARTer Than You Think (NeurIPS 2026 · similarity 0.86)
• MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encoding (NeurIPS 2024 · similarity 0.86)