Research map / Vision-language and multimodal / Human and embodied modeling
ExcluIR: Exclusionary Neural Information Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ExcluIR, a benchmark (3,452 human-checked) and training set (70,293) for exclusionary document retrieval, built from HotpotQA with ChatGPT-generated queries, and evaluates sparse, dense, and generative retrievers.
Problem
Neural retrieval models struggle to understand exclusionary queries, where users state what information they do not want in results; no benchmark existed for evaluating this in document retrieval.
Approach
Builds exclusionary queries from HotpotQA document pairs using ChatGPT generation, rephrasing, and manual correction, and measures whether a retriever ranks the desired positive document above the excluded negative document (via ΔR@N, ΔMRR, and RR metrics).
Key results
- Sparse and dense retrievers trained on HotpotQA or NQ320k score near random on ExcluIR (RR around 50-69%), with BM25 at 53.48% RR.
- Training SEAL on NQ320k plus ExcluIR training data raises ExcluIR RR from 60.02% to 78.02%, while NQ320k R@1 drops only about 1%.
- Generative models gained on average 17.75% RR from ExcluIR training versus 4.77% for dense models; human RR is 100%, so a gap remains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ExcluIR training data to SEAL trained on NQ320k (ablation) | SEAL · ExcluIR | RR: 60.02% vs. 78.02% | Helped |
| Add ExcluIR training data to retrieval models (ablation) | GENRE, SEAL, and NCI · ExcluIR | RR: 17.75% improvement | Helped |
| Add ExcluIR training data to retrieval models (ablation) | DPR, Sentence-T5, GTR, and ColBERT · ExcluIR | RR: 4.77% improvement | Helped |
| Add ExcluIR data to ColBERT trained on NQ320k (ablation) | ColBERT · ExcluIR | RR: 59.97 -> 59.71 | No clear effect |
| Train bi-encoder models on NQ320k then continue on HotpotQA instead of HotpotQA alone (ablation) | DPR and Sentence-T5 · ExcluIR | RR: 54.02 -> 54.66; 55.41 -> 56.82 | Helped |
| Expand training with NQ320k data (ablation) | ColBERT and SEAL · ExcluIR | RR: 55.57 -> 52.93; 55.52 -> 55.36 | Hurt |
| Increase model size from base to large (ablation) | DPR, Sentence-T5, GENRE, and NCI trained on HotpotQA · ExcluIR | RR: 54.02 -> 54.25; 55.41 -> 53.78; 52.10 -> 49.01; 50.93 -> 50.64 | Mixed |
| Increase model size from base to large with ExcluIR in training (ablation) | DPR, Sentence-T5, GENRE, and NCI · ExcluIR | RR: 61.19 -> 62.63; 66.75 -> 69.01; 69.07 -> 70.96; 73.75 -> 73.61 | Helped |
| Increase model size from base to large (ablation) | DPR, Sentence-T5, GENRE, and NCI trained on NQ320k · ExcluIR | RR: 58.76 -> 61.62; 67.83 -> 69.02; 58.65 -> 55.82; 56.84 -> 62.54 | Mixed |
| Use generative retrieval rather than sparse or dense retrieval | Retrieval models trained with ExcluIR data · ExcluIR | RR | Helped |
| Use ColBERT late interaction for exclusionary queries | ColBERT · ExcluIR | RR | Hurt |
Surprising findings
- Scaling model size or adding more training domains does not reliably improve exclusionary retrieval.
- ColBERT, despite strong standard benchmark scores, performs worst among neural models after ExcluIR fine-tuning because token-level matching overweights lexical overlap with excluded content.
Limitations
- Large gap remains between models and human performance (100% RR).
- Only single-round explicit exclusions are considered; implicit multi-round exclusions are not covered.
- Training queries are ChatGPT-generated from HotpotQA pairs, so the data depends on a proprietary generator.
Experimental setup
Datasets: HotpotQA NQ320k ExcluIR training set
Benchmarks: ExcluIR HotpotQA NQ320k
Metrics: R@1 R@5 R@10 MRR ΔR@1 ΔMRR RR
Baselines: BM25 DocT5Query DPR Sentence-T5 GTR ColBERT GENRE SEAL NCI
Builds on: HotpotQA DocT5Query ColBERT SEAL NCI
Closest papers by meaning
- SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction (ICLR 2025 · similarity 0.86)
- Absence Bench: Language Models Can’t See What’s Missing (NeurIPS 2025 · similarity 0.86)
- Making Retrieval-Augmented Language Models Robust to Irrelevant Context (ICLR 2024 · similarity 0.85)
- A Theory for Token-Level Harmonization in Retrieval-Augmented Generation (ICLR 2025 · similarity 0.85)
- Retrieval Head Mechanistically Explains Long-Context Factuality (ICLR 2025 · similarity 0.84)
- Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.