Research map / Large language models / Retrieval and knowledge
FinBen: A Holistic Financial Benchmark for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FinBen, an open-source holistic financial LLM benchmark spanning 24 tasks across seven aspects (IE, TA, QA, TG, RM, FO, DM), with three new datasets, and evaluates 15 LLMs including GPT-4, ChatGPT and Gemini.
Problem
Existing financial LLM benchmarks cover few tasks and mostly NLP tasks, leaving forecasting, risk management, text generation and decision-making (e.g., stock trading) under-evaluated for LLMs.
Approach
A broad financial evaluation taxonomy with seven aspects, zero-shot instruction-formatted evaluation of many open and closed LLMs, plus new datasets (EDTSum, FinTrade, Regulations) and an agent-based stock-trading evaluation built on FinMem.
Key results
- GPT-4 achieves the highest stock-trading Sharpe Ratio (1.51) and cumulative return (28.19%) across 10 stocks, versus Buy & Hold at 0.02 and -4.00%.
- Forecasting (stock movement) results for all LLMs are near random, with GPT-4 and Gemini only slightly above chance; MCC values are close to 0.
- Gemini leads on EDTSum summarization (Rouge-1 0.39) and GPT-4 leads most IE and NER tasks, e.g., NER entity F1 0.83.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4 instead of the strongest reported alternative | Financial named entity recognition · NER | EntityF1: 0.77 -> 0.83 | Helped |
| Instruction-tune the general model for financial tasks (FinMA 7B vs LLaMA2-7B-chat) | Financial sentiment analysis · FPB | F1: 0.39 -> 0.88 | Helped |
| Use instruction-tuned FinMA 7B rather than LLaMA2-7B-chat | Financial question answering · FinQA | EmAcc: 0.00 -> 0.04 | Helped |
| Fine-tune InternLM 7B on Chinese financial data to obtain CFGPT sft-7B-Full | English financial text classification · MultiFin | MicroF1: 0.34 -> 0.05 | Hurt |
| Use GPT-4 rather than Buy & Hold | Stock trading · FinTrade, 10 stocks | SR: 0.02 ± 0.87 -> 1.51 ± 1.08 | Helped |
| Use GPT-4 rather than GPT-4o | Stock trading · FinTrade, 10 stocks | SR: -0.19 ± 0.84 -> 1.51 ± 1.08 | Helped |
Surprising findings
- Fine-tuned financial model FinMA-7B underperforms general LLaMA2-7B-chat on several non-sentiment textual analysis tasks.
- CFGPT (Chinese financial SFT) shows performance declines versus its base InternLM-7B on English datasets such as MultiFin.
Limitations
- Restricted dataset sizes may limit generalization of financial understanding.
- Tasks are based on American market data and English texts, limiting global applicability.
- Computational constraints limited evaluation of larger models beyond LLaMA 70B.
Experimental setup
Datasets: NER FiNER-ORD FinRED SC CD FNXL FSRL FPB FiQA-SA TSA Headlines FOMC
Benchmarks: FinBen
Metrics: F1 Accuracy MCC EM Accuracy Entity F1 ROUGE BERTScore BARTScore RMSE Cumulative Return
Baselines: GPT-4 ChatGPT Gemini LLaMA2-70B FinMA-7B FinGPT DISC-FinLLM CFGPT DQN PPO A2C Buy & Hold
Builds on: PIXIU FinMem FLUE
Closest papers by meaning
- FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning (ICLR 2026 · similarity 0.91)
- FinRpt: Dataset, Evaluation System and LLM-based Multi-agent Framework for Equity Research Report Generation (AAAI 2026 · similarity 0.89)
- FinMMDocR: Benchmarking Financial Multimodal Reasoning with Scenario Awareness, Document Understanding, and Multi-Step Computation (AAAI 2026 · similarity 0.88)
- STEER-ME: Assessing the Microeconomic Reasoning of Large Language Models (NeurIPS 2025 · similarity 0.88)
- Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking (NeurIPS 2025 · similarity 0.87)
- FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.