Research map / Large language models / Agents, code and math
EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements
Issa Sugiura, Takashi Ishida, Taro Makino, Chieko Tazuke, Takanori Nakagawa, Kosuke Nakago, David Ha
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Text block to input (Summary+BS+CF+PL -> +Text) (ablation) | LLMs on EDINET-Bench · Accounting fraud detection | ROC-AUC: Claude 3.5 Sonnet 0.63 -> 0.73; Claude 3.5 Haiku 0.60 -> 0.67 | Improved |
| add Text block to input (ablation) | LLMs on EDINET-Bench · Earnings forecasting | ROC-AUC: Claude 3.7 Sonnet 0.58 -> 0.61; GPT-5 0.62 -> 0.65; Claude 3.5 Sonnet 0.55 -> 0.52 | No clear effect |
| expand input from Summary to Summary+BS+CF+PL (ablation) | LLMs on EDINET-Bench · Industry prediction | accuracy: Claude 3.5 Sonnet 0.24 -> 0.41; GPT-5 0.21 -> 0.31 | Improved |
| expand input from Summary to Summary+BS+CF+PL (ablation) | LLMs on EDINET-Bench · Accounting fraud detection | ROC-AUC: Claude 3.5 Sonnet 0.64 -> 0.63; Claude 3.7 Sonnet 0.59 -> 0.58; GPT-5 0.56 -> 0.62 | No clear effect |
| use zero-shot frontier LLM instead of Random Forest | Claude 3.5 Sonnet · Accounting fraud detection | ROC-AUC: 0.75 (Random Forest, Summary) vs 0.73 (Claude 3.5 Sonnet, with Text) | Hurt |
| use GPT-5 instead of older LLMs | GPT-5 · Earnings forecasting | ROC-AUC: 0.65 for GPT-5 vs 0.61 Claude 3.7 Sonnet | Improved |
| SFT Llama-3.2-1B on train split with Text input (ablation) | Llama-3.2-1B · Accounting fraud detection | ROC-AUC: 0.66 (BS+CF+PL) -> 0.54 (+Text) | Hurt |
| test year-wise performance for contamination (ablation) | LLMs · Accounting fraud detection by fiscal year | ROC-AUC | No clear effect |
Closest papers by meaning
- FinBen: A Holistic Financial Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.90)
- FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning (ICLR 2026 · similarity 0.87)
- EnterpriseBench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance (NeurIPS 2026 · similarity 0.86)
- Detecting Bugs with Substantial Monetary Consequences by LLM and Rule-based Reasoning (NeurIPS 2024 · similarity 0.86)
- FinMMDocR: Benchmarking Financial Multimodal Reasoning with Scenario Awareness, Document Understanding, and Multi-Step Computation (AAAI 2026 · similarity 0.86)
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
