Research map / Large language models / Retrieval and knowledge
HawkBench: Investigating Resilience of RAG Methods on Stratified Information-Seeking Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HawkBench, a 1,600-sample multi-domain RAG benchmark stratified into four explicit/implicit factoid/rationale query levels, and evaluates vanilla, enhanced, global RAG and long-LLM baselines on it.
Problem
Existing RAG benchmarks focus on isolated query types (mostly factoid), use heterogeneous or Wikipedia-like corpora, and fail to assess resilience across diverse information-seeking tasks.
Approach
Stratify information-seeking queries into four levels (explicit/implicit intent × factoid/rationale) over a shared multi-domain corpus of 229 long texts, with hybrid LLM-generated and human-annotated QA pairs, and score rationale answers with a sentence-level LLM-judged S-F1 metric.
Key results
- Enhanced RAG (RQRAG, 73.6 F1) and HyDE lead on Level-1 factoid queries, while MemoRAG reaches 34.1 S-F1 on Level-3 and 35.0 S-F1 on Level-4, the best among the methods reported.
- Standard vanilla RAG is far faster (about 0.6 to 1.7 queries per second) than long LLM (29 to 52 seconds per query under the reported units), and GraphRAG graph construction takes up to half an hour.
- Commercial LLMs such as GPT-4.1 score far below the best RAG system on Level-1 and Level-2 factoid queries (e.g. Legal Level-1: 15.3 vs 79.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase retrieved passages from top-5 to top-10 (ablation) | Vanilla RAG · HawkBench Level-1 | F1: 57.5 -> 71.3 | Helped |
| Increase retrieved passages from top-5 to top-50 (ablation) | Vanilla RAG · HawkBench Level-1 | F1: 57.5 -> 28.4 | Hurt |
| Increase retrieved passages from top-5 to top-10 (ablation) | Vanilla RAG · HawkBench Level-2 | F1: 38.6 -> 44.4 | Helped |
| Increase retrieved passages from top-5 to top-50 (ablation) | Vanilla RAG · HawkBench Level-2 | F1: 38.6 -> 22.8 | Hurt |
| Increase retrieved passages from top-5 to top-10 (ablation) | Vanilla RAG · HawkBench Level-3 | S-F1: 27.3 -> 33.5 | Helped |
| Increase retrieved passages from top-5 to top-50 (ablation) | Vanilla RAG · HawkBench Level-3 | S-F1: 27.3 -> 30.4 | Helped |
| Increase retrieved passages from top-5 to top-50 (ablation) | Vanilla RAG · HawkBench Level-4 | S-F1: 18.3 -> 34.7 | Helped |
| Use global RAG (MemoRAG) rather than standard RAG | RAG · HawkBench Level-3 | S-F1: 27.3 -> 34.1 | Helped |
| Use global RAG (MemoRAG) rather than standard RAG | RAG · HawkBench Level-1 | F1: 57.5 -> 50.2 | Hurt |
| Use global RAG (MemoRAG) rather than standard RAG | RAG · HawkBench Level-4 | S-F1: 18.3 -> 35.0 | Helped |
| Use global RAG (MemoRAG) rather than standard RAG | RAG · HawkBench Level-2 | F1: 38.6 -> 37.3 | Hurt |
| Use RAG rather than a long-context LLM | Qwen2.5-7B-Instruct · HawkBench Level-1 | F1: 12.9 -> 57.5 | Helped |
Surprising findings
- Global RAG methods underperform on factoid Level-1 and Level-2 queries despite excelling on rationale queries.
- Commercial LLMs remain competitive with RAG on Level-3 and Level-4 rationale tasks, though they lag badly on factoid queries.
Limitations
- Possible dataset bias from multi-domain curation and LLM-assisted generation errors.
- Corpora may have been seen during LLM pretraining, so memorization can inflate scores.
- Limited to representative methods and models; commercial RAG systems not evaluated.
Experimental setup
Datasets: books-3-textbooks legal contracts arXiv papers financial reports (SEC) novels
Benchmarks: HawkBench
Metrics: Rouge-L lexical F1 S-F1 latency (queries per second)
Baselines: Vanilla RAG HyDE RQRAG MemoRAG GraphRAG Long LLM Lingua-2 MInference gemini-2.5-flash gpt-4o-mini gpt-4.1
Builds on: HyDE RQ-RAG MemoRAG GraphRAG Lingua-2 MInference
Closest papers by meaning
- RAGGED: Towards Informed Design of Scalable and Stable RAG Systems (ICML 2025 · similarity 0.87)
- GISA: A Benchmark for General Information-Seeking Assistant (NeurIPS 2026 · similarity 0.86)
- InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents (ICLR 2026 · similarity 0.86)
- RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation (NeurIPS 2024 · similarity 0.86)
- Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievals (NeurIPS 2025 · similarity 0.86)
- Adapting to Non-Stationary Environments: Multi-Armed Bandit Enhanced Retrieval-Augmented Generation on Knowledge Graphs (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.