Research map / Large language models / Retrieval and knowledge
BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BRIGHT, a 1,384-query retrieval benchmark across 12 datasets (StackExchange, coding, theorem-based math) requiring reasoning beyond keyword/semantic matching, and shows that top retrievers score low while LLM-generated reasoning queries improve BM25.
Problem
Existing retrieval benchmarks rely on information-seeking queries where keyword or semantic matching suffices, so they do not test retrieval requiring intensive reasoning to determine query-document relevance.
Approach
Builds a retrieval benchmark from realistic user queries (StackExchange posts with cited web pages, coding problems, rewritten theorem questions) where positives share underlying reasoning, principles, algorithms, or theorems with the query, and hard negatives are topically similar; then evaluates retrievers with and without LLM-generated reasoning traces as queries.
Key results
- Best retriever Qwen (gte-Qwen1.5-7B) reaches 22.5 average nDCG@10 across 12 datasets; SFR-Embedding-Mistral reports 18.3 average on BRIGHT versus 59.0 on MTEB retrieval.
- LLM-generated reasoning queries improve retrieval by up to 12.2 points on average, with BM25 benefiting most.
- Retrieval augmentation with the top retriever raises StackExchange QA score by 1.9 points, while oracle documents raise it by 4.1 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLM-generated reasoning steps as retrieval queries instead of the original query | retrieval models on BRIGHT · BRIGHT | nDCG@10: up to 12.2 points | Helped |
| Use GritLM-generated reasoning steps as the retrieval query | BM25 on BRIGHT · BRIGHT | nDCG@10 | Helped |
| Use GritLM-generated reasoning steps as the retrieval query | retrieval models other than BM25 on BRIGHT · BRIGHT | nDCG@10 | No clear effect |
| Augment Claude-3.5-sonnet with Qwen-retrieved documents instead of using no retriever | Claude-3.5-sonnet · StackExchange QA | QA score: +1.9 points | Helped |
| Augment Claude-3.5-sonnet with oracle documents instead of using no retriever | Claude-3.5-sonnet · StackExchange QA | QA score: +4.1 points | Helped |
| Rerank BM25 results with Gemini-1.0 | BM25 on BRIGHT · BRIGHT | nDCG@10: +1.4 | Helped |
| Rerank BM25 results with GPT-4 | BM25 on BRIGHT · BRIGHT | nDCG@10: +3.1 | Helped |
| Rerank retrieved documents with the MS MARCO-trained MiniLM cross-encoder | retrieval models on BRIGHT · BRIGHT | nDCG@10 | Hurt |
| Continue training GritLM on StackExchange data without query-document mappings (ablation) | GritLM · StackExchange | average nDCG@10: 20.5 -> 20.4 | No clear effect |
Surprising findings
- Continued training of GritLM on in-domain StackExchange data does not improve BRIGHT performance (20.5 to 20.4 average).
- Cross-encoder MiniLM reranking trained on MS MARCO degrades retrieval quality as more documents are reranked.
Limitations
- Some StackExchange positives and negatives may be subjective despite annotator agreement
- Theorem-based queries are rewritten by GPT-4 and manually edited, so they may not reflect natural queries
- Small number of queries per dataset (76 to 194), making scores noisy
Experimental setup
Datasets: StackExchange posts and cited web pages LeetCode CodeSearchNet Pony documentation GSM8K GSM8K-RFT MATH AQuA-RAT TheoremQA CAMEL-Math MathInstruct ProofWiki
Benchmarks: BRIGHT
Metrics: nDCG@10 recall@1 QA accuracy (LLM-judged)
Baselines: BM25 BGE Instructor-Large Instructor-XL SBERT E5-Mistral SFR-Embedding-Mistral GritLM gte-Qwen1.5-7B Cohere embed v3 OpenAI text-embedding-3-large Voyage
Builds on: BEIR MTEB TheoremQA chain-of-thought prompting BM25
Closest papers by meaning
- Retriever-Free Retrieval-Augmented Reasoning via Corpus-Traversing MCTS (NeurIPS 2026 · similarity 0.89)
- FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents (NeurIPS 2025 · similarity 0.87)
- ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models (NeurIPS 2026 · similarity 0.87)
- Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Data (ICLR 2025 · similarity 0.87)
- Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation (ICLR 2025 · similarity 0.87)
- Retro*: Optimizing LLMs for Reasoning-Intensive Document Retrieval (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.