Research map / Multi-agent systems and games / LLM agents

AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AgentSelect, a benchmark that converts heterogeneous LLM leaderboard and tool-use artifacts into positive-only query-to-agent interaction data over (model, toolset) capability profiles, and benchmarks recommenders on it.

Problem

Selecting a deployable LLM agent configuration (backbone model plus tools) for a free-form narrative query lacks query-conditioned supervision over complete configurations; existing leaderboards and tool benchmarks evaluate components in isolation.

Approach

Reframes agent selection as narrative query-to-agent recommendation over (M,T) capability profiles and unifies heterogeneous evaluation artifacts into positive-only interactions, adding LLM/tool-retriever-synthesized compositional pseudo-positives (Part III); finds content-aware matching beats ID-based CF/GNN under long-tail supervision.

Key results

• On Part III, TwoTower (BGE-M3) reaches nDCG@10 0.8501 and MRR 0.8654, while popularity-style graph CF methods (NGCF, LightGCN) score near 0 on Part III.
• On MuleRun, tuned EasyRec* improves Top-1 precision from 0.3556 to 0.4000 and nDCG@10 from 0.5491 to 0.5870 versus untuned EasyRec.
• Counterfactual edits show adding an irrelevant tool lowers score (mean Δs 0.0825) with 84% consistency, while adding a redundant tool has near-zero effect.

Surprising findings

• Adding Part III to Parts I+II drops Part I nDCG@10 from 0.9457 to 0.7111 even though it improves Part III, which the authors attribute to distribution shift between regimes.
• Removing backbone ID barely changes performance, suggesting the ranker relies on textual capability content rather than memorized IDs.

Limitations

• Part III pseudo-positives are synthesized and validated by an LLM judge rather than by end-to-end execution, so they are not certified optimal.
• Main evaluation is retrieval-based; end-to-end task completion is only checked on a small deployed-agent study.
• OneRec is a simplified variant, and generative recommenders over-select training-seen agents under sparse positives.

Experimental setup

Datasets: Open LLM Leaderboard MMLU BBH ToolHop ToolGen UltraTool APIBank GAIA

Benchmarks: AgentSelect MuleRun

Metrics: Precision@10 Recall@10 F1@10 nDCG@10 MRR@10 Hit@10

Baselines: MF LightFM NCF NGCF KGAT LightGCN SimGCL BGE-Reranker KaLM-v2.5 EasyRec OneRec

Builds on: LightGCN EasyRec OneRec StableToolBench

Closest papers by meaning

• AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems (NeurIPS 2025 · similarity 0.91)
• MoRAgent: Parameter Efficient Agent Tuning with Mixture-of-Roles (ICML 2025 · similarity 0.88)
• AgentSquare: Automatic LLM Agent Search in Modular Design Space (ICLR 2025 · similarity 0.88)
• AgentSwift: Efficient LLM Agent Design via Value-Guided Hierarchical Search (AAAI 2026 · similarity 0.87)
• WideSearch: Benchmarking Agentic Broad Info-Seeking (ICLR 2026 · similarity 0.87)
• ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents (ICLR 2025 · similarity 0.87)