Research map / Label-efficient and robust learning / Self-supervised and contrastive
When is an Embedding Model More Promising than Another?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an information-sufficiency score, derived from Le Cam deficiency and statistical experiment comparison, to rank embedding models pairwise without labeled downstream data, validated on NLP (MTEB) and molecular ADMET tasks.
Problem
Evaluating and ranking embedding models usually requires training downstream models on labeled tasks, which is costly and not scalable; a standardized, label-free way to compare embedders is lacking.
Approach
Casts embedder comparison as comparing statistical experiments: uses sufficiency, informativeness and Le Cam's deficiency, and makes them tractable via an information sufficiency (IS) score estimated with learned Gaussian-mixture Markov kernels on unlabeled data, taking the median IS of each embedder against all others as its rank score.
Key results
- IS score correlates with MTEB average performance with Spearman 0.90 and Pearson 0.94 across 56 NLP tasks.
- On 31 ADMET molecular tasks, IS ranking achieves Spearman 0.95 and Kendall-tau 0.80.
- Embedders are clustered by Louvain communities consistent with architecture and training objective (e.g., LLM-based, BERT-based).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the information sufficiency score to rank text embedders | Text embedding models · MTEB | Spearman correlation with average benchmark score: 0.90 | Helped |
| Use the information sufficiency score to rank text embedders | Text embedding models · MTEB | Pearson correlation with average benchmark score: 0.94 | Helped |
| Use the information sufficiency score to rank molecular embedders | Molecular embedding models · ADMET tasks | Spearman ranking correlation: 0.95 | Helped |
| Use the information sufficiency score to rank molecular embedders | Molecular embedding models · ADMET tasks | Kendall-Tau coefficient: 0.80 | Helped |
| Use the information sufficiency score to rank molecular embedders | Molecular embedding models · Molecular downstream tasks (31 tasks) | Spearman ranking correlation: 0.94 | Helped |
| Increase molecular model size to GPT-1.2B | GPT molecular embedder · ZINC 250k | Information sufficiency score | Helped |
| Use attribute masking or context prediction objectives for 2D-GNN pretraining | Molecular 2D-GNN embedders · ADMET tasks | Downstream task performance | Hurt |
Surprising findings
- Some molecular embedders are ranked below a randomly initialized, untrained GNN baseline by the IS score, and these also perform worse downstream.
- Two-dimensional GNN pretraining objectives (attribute masking, context prediction) are among the least informative, and masked-LM string models fail to simulate others except the largest GPT model.
Limitations
- Effectiveness depends on the number and diversity of embedders in the pool.
- Proxy for deficiency relies on learned Gaussian-mixture Markov kernels rather than directly minimizing total variation or estimating f-divergences.
Experimental setup
Datasets: MTEB datasets (banking77, sickr-sts, amazon_polarity, snli, imdb, ag_news, sst2, tweet_eval, paws-x, rotten_tomatoes) ZINC 250k Therapeutic Data Commons ADMET tasks (31 tasks)
Benchmarks: MTEB Therapeutic Data Commons ADMET
Metrics: Spearman correlation Pearson correlation Kendall-tau
Baselines: MTEB leaderboard downstream performance randomly initialized GNN
Builds on: Le Cam deficiency Blackwell sufficiency Körner-Marton less noisy ordering MTEB
Closest papers by meaning
- Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.88)
- Analyzing Similarity Metrics for Data Selection for Language Model Pretraining (NeurIPS 2025 · similarity 0.88)
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (ICLR 2025 · similarity 0.87)
- EmbedLLM: Learning Compact Representations of Large Language Models (ICLR 2025 · similarity 0.86)
- Scaling Embedding Layers in Language Models (NeurIPS 2025 · similarity 0.86)
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.