Research map / Vision-language and multimodal / Multimodal LLMs

FinMMDocR: Benchmarking Financial Multimodal Reasoning with Scenario Awareness, Document Understanding, and Multi-Step Computation

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces FinMMDocR, a bilingual benchmark of 1,200 expert-annotated financial numerical reasoning questions over long multimodal documents, testing MLLMs and RAG pipelines on scenario awareness, document understanding, and multi-step computation.

Problem

Existing financial QA and document QA benchmarks lack implicit real-world financial scenarios, diverse visually-rich long financial documents, and precise multi-step numerical computation, so they do not reflect expert financial reasoning with MLLMs.

Approach

A benchmark pairing implicit financial scenarios, 837 long bilingual financial documents (avg 50.8 pages), and multi-step extraction-plus-calculation questions with exact-answer, 0.2%-tolerance evaluation, with PoT prompting and RAG comparisons.

Key results

• Best MLLM OpenAI o4-mini-high reaches 58.00% accuracy under image input with PoT, the top score in the benchmark.
• Best text-input OCR+LLM model Gemini 2.5 Pro Preview reaches 53.83%, while open-source models perform far lower (e.g., Qwen2.5-VL 72B at 12.92% with images).
• Vision RAG methods outperform text-only retrieval, with ColQwen2.5 achieving the best RAG performance, and Agentic RAGs underperform it despite higher token and time costs.

Surprising findings

• Complex Agentic RAG pipelines underperform the simpler ColQwen2.5 retrieval baseline despite consuming more tokens and time.
• OpenAI o4-mini-high is the only model whose image-input accuracy exceeds its OCR-text counterpart.

Limitations

• No model exceeds 60% accuracy, indicating the benchmark remains far from solved.
• Open-source MLLMs perform poorly, and image-input performance lags OCR+LLM for most models.
• Error analysis is based on 100 sampled failures from one model only.

Experimental setup

Datasets: DocMath-Eval CompLong (re-annotated 600 English questions) 385 Chinese research reports (new)

Benchmarks: FinMMDocR

Metrics: accuracy (0.2% tolerance)

Baselines: OpenAI o4-mini-high Gemini 2.5 Pro Preview Claude 3.7 Sonnet Doubao-1.5-vision-pro GPT-4o DeepSeek-R1 DeepSeek-V3 Qwen2.5-VL 72B ColQwen2.5 ViDoRAG MDocAgent SimpleDoc

Builds on: DocMath-Eval Program-of-Thought ColQwen2.5 ViDoRAG VRAG-RL

Closest papers by meaning

• FinBen: A Holistic Financial Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.88)
• FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning (ICLR 2026 · similarity 0.87)
• BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset (NeurIPS 2025 · similarity 0.86)
• MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.86)
• MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.85)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.85)