Research map / Label-efficient and robust learning / Recognition and long-tail

ModelLens: Finding the Best for Your Task from Myriads of Models

NeurIPS 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Data augmentation and generalization bias · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ModelLens, a metric-aware ranking model trained on a 1.62M-record leaderboard interaction benchmark that recommends models for unseen datasets and models without running candidates on the target dataset.

Problem

Choosing the best pretrained model for a new dataset is infeasible at open-ecosystem scale: AutoML and routing use small fixed pools, and transferability estimation needs per-candidate forward passes on the target dataset.

Approach

Learn a performance-aware latent space from leaderboard (model, dataset, metric, performance) tuples with a structural prior over model size and family plus a residual interaction term, using ID-dropout so unseen models and datasets can be ranked from metadata and descriptions alone.

Key results

• On performance completion, ModelLens reaches weighted Kendall tau 0.868 vs 0.763 for ZAP; on new datasets it reaches 0.745 vs 0.253 for ZAP.
• Recommended Top-K model pools improve KNNRouter by 81.1% average exact match (0.227 to 0.411) and Router-R1-Qwen from 0.355 to 0.482 average.
• On the 8 vision transferability benchmarks, ModelLens (feature-free) averages tau_w 0.369 and with feature augmentation 0.609, best on every dataset.

Surprising findings

• Feature-free ModelLens (0.369 average tau) is outperformed by the feature augmented version (0.609), and the authors note learned-space clustering is not recovered by description-only embeddings.
• Raw description similarity retrieves performance-irrelevant models (e.g., DeBERTa-MNLI) for MMLU, showing textual similarity is a poor proxy for performance.

Limitations

• Feature-free performance on seen-dataset vision transfer is weaker than forward-pass augmented variant
• Leaderboard records mix evaluation protocols (zero-shot, fine-tuning, prompting) and are noisy

Experimental setup

Datasets: HuggingFace Model Hub leaderboard records Open LLM Leaderboard PapersWithCode NQ PopQA HotpotQA Musique Bamboogle Aircraft Cars DTD Pets

Benchmarks: Model Recommendation in the Wild benchmark NQ PopQA HotpotQA Musique Bamboogle Aircraft Cars DTD Pets Flowers102 Food101

Metrics: Kendall weighted tau NDCG@k Hit@k Recall@k Exact Match MRR

Baselines: ZAP Task2Vec Model Size Model Popularity H-Score NCE LEEP NLEEP LogME PACTran OTCE LFC

Builds on: BPR Plackett-Luce listwise ranking DropoutNet Router-R1

Closest papers by meaning

• From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder Pipeline (ICML 2025 · similarity 0.85)
• RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.84)
• Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.84)
• DataDecide: How to Predict Best Pretraining Data with Small Experiments (ICML 2025 · similarity 0.84)
• LiveXiv - A Multi-Modal live benchmark based on Arxiv papers content (ICLR 2025 · similarity 0.83)
• We Should Chart an Atlas of All the World's Models (NeurIPS 2025 · similarity 0.83)