Research map / Large language models / Retrieval and knowledge
UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces UDA, a benchmark of 2,965 raw real-world documents (PDF/HTML) with 29,590 expert-annotated Q&A pairs, and evaluates parsing, retrieval, long-context, chain-of-thought/code, and end-to-end RAG pipelines across LLMs.
Problem
Existing document QA benchmarks use clean, segmented, or Wikipedia-only inputs, so they do not capture the difficulties of RAG over raw, unstructured, lengthy finance, academic, and web documents.
Approach
A benchmark that keeps documents in their native PDF/HTML formats (unparsed, unsegmented) and pairs them with Q&A from six sub-datasets, enabling granular evaluation of each RAG component plus end-to-end comparison with long-context LLMs.
Key results
- End-to-end, GPT-4-Turbo scored 45.7 average (rule-based) across six UDA sub-datasets, the top result among eight LLMs.
- On FinHybrid, long-context GPT-4-Turbo scored 37.4 vs 43.4 with RAG top-5; for Qwen-1.5-7B long-context dropped to 3.0 vs 21.0 with RAG.
- Relative LCS retrieval: OpenAI text-embedding-3-large led on most datasets, but BM-25 beat it on FinHybrid at @1 (65.6 vs 57.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4-Omni table parsing instead of raw-text extraction (ablation) | GPT-4-Turbo on Tabular FinHybrid · Tabular FinHybrid | Exact Match: 68.0 -> 72.4 | Helped |
| Use well-parsed tables instead of raw-text extraction (ablation) | Llama-3-8B on Tabular FinHybrid · Tabular FinHybrid | Exact Match: 51.6 -> 59.5 | Helped |
| Use GPT-4-Omni table parsing instead of raw-text extraction (ablation) | GPT-4-Turbo on PaperTab · PaperTab | F1: 42.4 -> 44.3 | Helped |
| Use CV-based table parsing rather than raw-text extraction (ablation) | GPT-4-Turbo on Tabular FinHybrid · Tabular FinHybrid | Exact Match: 68.0 -> 61.3 | Hurt |
| Use CV-based parsing followed by LLM table conversion rather than raw-text extraction (ablation) | GPT-4-Turbo on Tabular FinHybrid · Tabular FinHybrid | Exact Match: 68.0 -> 52.4 | Hurt |
| Replace OpenAI-retrieved context with human-annotated evidence (ablation) | Llama-3-8B on FinHybrid · FinHybrid | Exact Match: 37.9 -> 51.0 | Helped |
| Replace OpenAI-retrieved context with human-annotated evidence (ablation) | GPT-4-Turbo on FinHybrid · FinHybrid | Exact Match: 45.9 -> 69.4 | Helped |
| Replace retrieved chunks with long-context input (ablation) | Qwen-1.5-7B on FinHybrid · FinHybrid | Exact Match: 21.0 -> 3.0 | Hurt |
| Replace retrieved chunks with long-context input (ablation) | GPT-4-Turbo on FinHybrid · FinHybrid | Exact Match: 43.4 -> 37.4 | Hurt |
| Add Chain-of-Thought prompting (ablation) | Llama-3-8B on FinHybrid · FinHybrid | Exact Match: 21.3 -> 37.9 | Helped |
| Add Chain-of-Thought prompting (ablation) | GPT-4-Turbo on FinHybrid · FinHybrid | Exact Match: 45.9 -> 45.9 | No clear effect |
| Use code-generation strategy instead of basic generation (ablation) | GPT-4-Turbo on FinHybrid · FinHybrid | Exact Match: 45.9 -> 32.2 | Hurt |
Surprising findings
- Raw text extraction performed decently against GPT-4-Omni and well-parsed tables, while classic CV parsing pipelines underperformed.
- Llama-3-8B with human-annotated context (51.0) outperformed GPT-4 with retrieved chunks (45.9) on FinHybrid, suggesting retrieval quality matters more than model scale.
Limitations
- Parsing quality is assessed only via downstream QA, not by direct comparison of parsed content.
- Noise sensitivity and hallucination were not analyzed.
- Long-context experiments used only a 600-document subset due to cost.
Experimental setup
Datasets: FinQA TAT-DQA Qasper FetaQA Natural Questions
Benchmarks: UDA (FinHybrid, TatHybrid, PaperTab, PaperText, FetaTab, NqText)
Metrics: exact match span-level F1 numeracy-focused F1 relative LCS LLM-score (0-4 normalized)
Baselines: BM-25 all-MiniLM-L6 all-mpnet-base text-embedding-3-large ColBERT GPT-4-Omni parsing Yolox+Tesseract+TableTransformer CV parsing
Builds on: FinQA TAT-DQA Qasper FetaQA Natural Questions
Closest papers by meaning
- Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers (NeurIPS 2025 · similarity 0.89)
- FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents (NeurIPS 2025 · similarity 0.88)
- Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation (ICML 2024 · similarity 0.88)
- Benchmarking Large Language Models in Retrieval-Augmented Generation (AAAI 2024 · similarity 0.87)
- A Theory for Token-Level Harmonization in Retrieval-Augmented Generation (ICLR 2025 · similarity 0.87)
- RAGGED: Towards Informed Design of Scalable and Stable RAG Systems (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.