Research map / Label-efficient and robust learning / Recognition and long-tail

Capturing LLM Capabilities via Evidence-Calibrated Query Clustering

NeurIPS 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Contrastive representation theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ECC, a query clustering method that jointly uses query embeddings and limited pairwise model comparisons (Bradley-Terry capability profiles with soft responsibilities) to build capability-aware clusters for query-specific LLM ranking and routing.

Problem

Semantic or human-taxonomy query clusters often fail to match latent LLM capability demands, so cluster-level capability estimates generalize poorly to unseen queries; LLM-judge-based capability clustering is costly and yields hard partitions.

Approach

Calibrates prior semantic centroids with posterior pairwise model comparisons via alternating optimization of entropy-regularized soft responsibilities, where each cluster has a Bradley-Terry capability profile; at inference a single random probe comparison plus the query embedding yields a query-specific mixture ranking.

Key results

• ECC improves ranking quality gains over human-labeled clustering by 17.64 pp and over embedding-only clustering by about 18 pp on average across benchmarks (Emb+Probe-infer, K=30).
• On SPROUT with K=30, ECC reaches 26.15% ranking quality gain under Emb+Probe-infer versus 13.98% for embedding-only clustering.
• In guided query routing, ECC improves response quality of routed queries by 16.6% on average over embedding-only clustering.

Surprising findings

• Human-defined taxonomies yield only marginal gains over a global BT model and even degrade quality on MMLU.
• Comparison-only clusters degrade sharply under embedding-only inference, showing a signal mismatch between clustering and inference signals.

Limitations

• Evaluated only on text-only queries and single standalone queries without conversational context.
• Performance gains depend on availability of pairwise comparison evidence per query for clustering.

Experimental setup

Datasets: SPROUT RouterBench Open LLM Leaderboard v2 MMLU MMLU-Pro MATH

Benchmarks: SPROUT RouterBench Open LLM Leaderboard v2 MMLU MMLU-Pro MATH

Metrics: ranking quality gain (per-query BT loss reduction, pp) Kendall-tau ARI NMI response quality for routing

Baselines: Human-labeled clustering Embedding-only clustering Comparison-only clustering Prompt-to-Leaderboard

Builds on: Bradley-Terry model K-Means Prompt-to-Leaderboard

Closest papers by meaning

• Bohdi: Heterogeneous LLM Fusion with Automatic Data Exploration (NeurIPS 2025 · similarity 0.85)
• GraphRouter: A Graph-based Router for LLM Selections (ICLR 2025 · similarity 0.85)
• SAGE: Scalable Automated Robustness Augmentation for LLM Knowledge Evaluation (NeurIPS 2026 · similarity 0.85)
• SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines (NeurIPS 2025 · similarity 0.84)
• Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.84)
• LLM Alignment as Retriever Optimization: An Information Retrieval Perspective (ICML 2025 · similarity 0.84)