Research map / Multi-agent systems and games / LLM agents
BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BankerToolBench, an open benchmark of 100 end-to-end junior investment banking workflows graded by banker-written rubrics and an agentic verifier (Gandalf), finding the best frontier model passes only 16% of tasks.
Problem
Existing AI benchmarks test isolated capabilities or simplified, single-output tasks, and do not measure whether agents can complete realistic end-to-end professional workflows such as junior investment banking work.
Approach
Banker-authored tasks with multi-file deliverables (Excel, PowerPoint, PDF, Word) and 100+ weighted binary rubric criteria per task, graded by an agent-as-a-judge verifier that can inspect formulas and files, packaged as an RL environment in the Harbor framework.
Key results
- Best model GPT-5.4 passes only 16% of tasks under the Pass Rate threshold; 0% of AI deliverables were rated sendable as-is by bankers.
- Verifier accuracy 88.2% vs human inter-rater agreement 84.6% against banker grades (kappa 0.76).
- Post-training with Dr. GRPO improves BTB performance by up to 13x for Qwen 3 4B and 5x for Qwen 3 32B.
Surprising findings
- Claude Opus 4.5 and 4.6 hard-code spreadsheet values instead of formula-backed cells, significantly lowering their scores.
- Providing banker domain and formatting context significantly improves agent performance, suggesting domain knowledge is a key gap.
Limitations
- Benchmark uses standardized public data (SEC filings, market data) rather than messy live deal information and lacks institutional context.
- Evaluates individuals on discrete workflows, not team collaboration or feedback cycles.
- US-centric task set.
Experimental setup
Datasets: BankerToolBench tasks (100)
Benchmarks: BankerToolBench
Metrics: weighted rubric Score Pass Rate Pass@1 Pass@3 Pass^3 tool error rate runtime
Baselines: GDPval APEX-Agents Finance Agent Benchmark
Builds on: Harbor OpenHands OpenCode GDPval APEX-Agents Dr. GRPO DPO
Closest papers by meaning
- EnterpriseBench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance (NeurIPS 2026 · similarity 0.91)
- FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning (ICLR 2026 · similarity 0.86)
- PaperBench: Evaluating AI’s Ability to Replicate AI Research (ICML 2025 · similarity 0.86)
- InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation (ICLR 2025 · similarity 0.86)
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation (ICLR 2026 · similarity 0.85)
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.