Research map / Large language models / Retrieval and knowledge

Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation

ICML 2024 · Large language models › Retrieval and knowledge · approach: RAG robustness and evaluation · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes automatically generating task-specific multiple-choice exams from a RAG corpus and scoring RAG pipelines with a hierarchical Item Response Theory model, showing retrieval choice can matter more than LLM size.

Problem

Evaluating Retrieval-Augmented LLM (RAG) pipelines on narrow, domain-specific tasks lacks an automated, interpretable, and cost-efficient method, since public benchmarks and existing RAG metrics do not capture task-specific factual accuracy.

Approach

An LLM generates multiple-choice exams from the task's document corpus, filtered by NLP-based a-priori checks; RAG pipelines are scored by answering the exam, and a hierarchical IRT model decomposes ability into LLM, retriever, and in-context-learning components, with item information used to iteratively prune uninformative questions.

Key results

• Best absolute accuracy on t_ops with LlamaV2-70B and BM25 or MultiQA reaches about 71-73%, versus 63% for Oracle-free retrieval gaps shown in the table (e.g. ClosedB 63.0%).
• IRT analysis on t_sec: switching from SIAM to DPRV2 yields a larger ability gain (about 1.61 logits) than switching from Mistral-7B to LlamaV2-70B (about 0.66), supporting retrieval choice over model size.
• Averaged over tasks, LlamaV2-70B with DPRV2 reaches 71.8% accuracy vs 49.3% for closed-book LlamaV2-70B.

Measured effects reported in the paper

Surprising findings

• A poorly aligned retriever (SIAM) can yield worse accuracy than no retrieval at all (ClosedB), e.g. on t_sec and t_ops.
• Oracle can be outperformed by some retrievers on t_arx, because multiple document chunks overlap in information.

Limitations

• Evaluation is limited to multiple-choice factual accuracy and does not measure all RAG dimensions such as faithfulness or coherence.
• Only three LLMs (Mistral-7B, LlamaV2-13B/70B), three ICL modes, and seven retrieval mechanisms were tested.
• Exam questions are generated by LlamaV2-70B itself, which may introduce generator bias.

Experimental setup

Datasets: AWS Knowledge Center webpages ArXiv abstracts StackExchange questions SEC filings

Benchmarks: DevOps (t_ops) Arxiv (t_arx) StackExchange (t_stk) SEC Filings (t_sec)

Metrics: accuracy IRT ability item information

Baselines: ClosedBook SIAM DPR BM25 MultiQA DPRV2 Oracle

Builds on: Item Response Theory BM25 Bloom's revised taxonomy

Closest papers by meaning

• Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.89)
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (ICLR 2024 · similarity 0.89)
• Benchmarking Large Language Models in Retrieval-Augmented Generation (AAAI 2024 · similarity 0.89)
• Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting (ICLR 2025 · similarity 0.89)
• Retrieval is Not Enough: Enhancing RAG through Test-Time Critique and Optimization (NeurIPS 2025 · similarity 0.89)
• Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers (NeurIPS 2025 · similarity 0.88)