Research map / Multi-agent systems and games / LLM agents

FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces FinSearchComp, a 635-question open-domain financial search benchmark (Global and Greater China subsets) with expert-curated tasks, and evaluates 21-22 web products and APIs with an LLM judge against human experts.

Problem

Existing financial benchmarks pre-supply context and bypass open-domain search, and browsing benchmarks lack domain-specific time-sensitive and reconciliation demands, so agents' realistic financial search ability is not measured.

Approach

An expert-curated, end-to-end open-domain financial search benchmark with three analyst-style task families (time-sensitive fetching, simple historical lookup, complex historical investigation), bilingual Global and Greater China subsets, rubric-based LLM-as-judge scoring with tolerance bands, and a human expert baseline.

Key results

• Grok 4 (web) scores 68.9% on the Global subset, the top model, still 6.1 pp below human experts (75.0%).
• On Greater China, DouBao (web) leads at 54.2% average, with all models more than 34 pp below human experts (88.3%).
• Search-augmented models gain on average 40.8, 29.0, and 8.1 points on T1, T2, and T3 over models without search.

Surprising findings

• Reasoning models show an average 7.0-point decline on time-sensitive T1 tasks, attributed to overthinking.
• No product exceeds 30 on T3 except Grok 4 and GPT-5-Thinking, largely because structured API/SQL retrieval is absent from web-search-only products.

Limitations

• LLM-as-judge agreement with humans is about 95%, with per-subset accuracies of 91.5% to 99.8%, leaving residual scoring error.
• Time-sensitive answers depend on market close timing and data-provider latency, complicating reproducibility.
• Many products are closed web interfaces, so results depend on undisclosed search tools and versions.

Experimental setup

Benchmarks: FinSearchComp

Metrics: 0-1 accuracy score human expert accuracy comparison

Baselines: Grok 4 GPT-5-Thinking Gemini-2.5-pro DouBao DeepSeek-R1 Qwen3-235B-A22B-2507 Kimi k2 ERNIE-X1 HunYuan-T1

Builds on: BrowseComp FinQA Finance Agent Benchmark

Closest papers by meaning

• FinBen: A Holistic Financial Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.91)
• FinRpt: Dataset, Evaluation System and LLM-based Multi-agent Framework for Equity Research Report Generation (AAAI 2026 · similarity 0.88)
• EnterpriseBench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance (NeurIPS 2026 · similarity 0.87)
• FinMMDocR: Benchmarking Financial Multimodal Reasoning with Scenario Awareness, Document Understanding, and Multi-Step Computation (AAAI 2026 · similarity 0.87)
• BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows (NeurIPS 2026 · similarity 0.86)
• FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making (NeurIPS 2024 · similarity 0.86)