Research map / Large language models / Retrieval and knowledge
FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds FreshStack, an automatic pipeline that turns recent Stack Overflow Q&A plus GitHub technical docs into nugget-level IR/RAG benchmarks across five niche topics, and evaluates retrievers, rerankers, and RAG generators on them.
Problem
Existing IR and RAG benchmarks are outdated, artificially easy, and not specialized, and they risk LLM contamination and overfitting. Realistic evaluation of retrieval over technical documents is needed.
Approach
Automatically build benchmarks from community-asked Stack Overflow questions with accepted answers, decompose answers into GPT-4o nuggets, pool documents from a multi-repository GitHub corpus using inference and oracle retrieval plus fusion, and judge nugget-level document support with GPT-4o.
Key results
- Oracle-setting Stack Overflow Nuggets with fusion reaches up to 0.601 alpha-nDCG@10 on Yolo v7 & v8 versus 0.390 or lower for individual inference-setting models, showing large headroom.
- Ensemble fusion of four retrievers outperforms each individual model on nearly all topics and metrics, e.g., LangChain inference-setting fusion alpha-nDCG@10 0.322 vs BM25 0.228 in pooling.
- Voyage rerank-2 helps BGE, Voyage and fusion on LangChain, Yolo and Godot4 but hurts on Laravel and Angular; GPT-4.1 with fusion+rerank and oracle nuggets gives best RAG A_strict (e.g., 0.717 on Laravel).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace an individual retriever with Fusion (4 models) | Document retrieval using only the Stack Overflow question · FreshStack topics | α-nDCG@10 | Helped |
| Add Voyage AI rerank-2 after BM25 | BM25 document retrieval · FreshStack topics | α-nDCG@10 | Helped |
| Add Voyage AI rerank-2 after the dense retriever | BGE (Gemma-2), Voyage-large-2, and Fusion · Laravel 10 & 11 and Angular 16, 17 & 18 | α-nDCG@10 | Hurt |
| Add Voyage AI rerank-2 after dense retrieval | Dense retrievers · LangChain, Yolo v7 & v8, and Godot4 | α-nDCG@10 | Helped |
| Use Stack Overflow nuggets instead of the question for retrieval | Fusion (4 models) · FreshStack topics | α-nDCG@10 | Helped |
| Use GPT-4o sub-questions instead of a closed-book answer for inference pooling | Inference-setting retrieval · LangChain, Laravel 10 & 11, Angular 16, 17 & 18, and Godot4 | α-nDCG@10 | Helped |
| Use GPT-4o nuggets instead of the answer for oracle pooling | Oracle-setting retrieval · LangChain, Yolo v7 & v8, Angular 16, 17 & 18, and Godot4 | α-nDCG@10 | Helped |
| Use GPT-4.1 instead of GPT-4o as the answer generator | RAG generation · FreshStack topics | All Strict (A_strict) | Helped |
| Use oracle retrieved context instead of inference-setting context | RAG generation · FreshStack topics except Yolo v7 & v8 | All Strict (A_strict) | Helped |
Surprising findings
- Reranking reduces retrieval quality for several dense retrievers and fusion on Laravel 10 & 11 and Angular 16/17/18.
- Fusion+rerank RAG with GPT-4.1 is competitive with or exceeds the oracle-context setting on Yolo v7 & v8.
Limitations
- Nugget support and nugget generation rely on GPT-4o, with human calibration only on 60 LangChain queries.
- Benchmark is susceptible to contamination since it is built from Stack Overflow data and newer LLMs have later knowledge cutoffs.
- Fusion is computationally expensive at inference time and repository selection is manual.
Experimental setup
Datasets: Stack Overflow XML data archive (Oct 2024) GitHub repositories for technical documents
Benchmarks: FreshStack (LangChain, Yolo v7 & v8, Laravel 10 & 11, Angular 16/17/18, Godot4)
Metrics: alpha-nDCG@10 Coverage@20 Recall@50 All Strict (A_strict) nugget recall
Baselines: BM25 BGE (Gemma-2) E5 Mistral (7B) Voyage-large-2 Fusion (4 models) Voyage rerank-2
Builds on: BEIR HyDE TREC RAG 2024 nugget evaluation Cranfield paradigm
Closest papers by meaning
- UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis (NeurIPS 2024 · similarity 0.88)
- Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation (ICML 2024 · similarity 0.88)
- BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval (ICLR 2025 · similarity 0.87)
- Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks (ICLR 2026 · similarity 0.87)
- Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers (NeurIPS 2025 · similarity 0.87)
- Autofocus Retrieval: An Effective Pipeline for Multi-Hop Question Answering With Semi-Structured Knowledge (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.