Research map / Large language models / Retrieval and knowledge

OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries

NeurIPS 2026 · Large language models › Retrieval and knowledge · approach: Explanatory and multiple-choice QA benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OBLIQ-Bench, five retrieval tasks where relevance depends on latent patterns (implicit stance, failure modes, proof strategy, authorship style, tip-of-the-tongue), showing reasoning-LLM tournament reranking far outperforms current retrievers.

Problem

Existing retrieval benchmarks are saturated because relevant documents are easy for strong retrievers to surface; efficient retrievers fail on "oblique" queries whose relevance depends on latent, non-surface document attributes that a reasoning LLM can verify.

Approach

Define obliqueness via a retrieval--verification gap (best reasoning-LLM reranker score minus best scalable retriever score), and build an LLM-driven pipeline (human-defined latent lens, corpus-wide attribute extraction, clustering, abstract query generation, pooling) to create five oblique benchmark tasks.

Key results

• On Twitter-Conflict, the best non-oracle system (GPT-5.2 Multi-Hop Agent) reaches .215 pooled NDCG@10 while the oracle GPT-5.2 Tournament reaches .436.
• On Congress Hearings, Gemini-Embedding-2 reaches only .079 Recall@10 while the oracle GPT-5.2 Tournament reaches .957 Recall@10 and .913 NDCG@10.
• On Writing-Style, Gemini-Embedding-2 is the best single-stage retriever at .164 NDCG@10 versus .515 for the oracle tournament.

Surprising findings

• Multi-hop agentic search and query rewriting with GPT-5.2 hurt substantially on Writing-Style, performing worse than plain BM25 or dense retrieval.
• The 149M-parameter LateOn outperforms every other single-stage retriever on Congress Hearings, including Gemini-Embedding-2.

Limitations

• Oracle reranker is not perfect and its cost is high, so the verification upper bound is approximate.
• Queries and relevance labels are largely LLM-generated and LLM-judged with expert oversight, which may introduce label bias.
• Only five tasks across a limited set of corpora and domains.

Experimental setup

Datasets: Twitter (X API full-archive) WildChat-4.8M Putnam and undergraduate math competitions American Mathematical Monthly GovInfo congressional hearing transcripts LWN.net LessWrong Quanta Magazine

Benchmarks: OBLIQ-Bench Twitter-Conflict OBLIQ-Bench WildChat Errors OBLIQ-Bench Math Meta-Program OBLIQ-Bench Writing-Style OBLIQ-Bench Congress Hearings

Metrics: NDCG@10 NDCG@50 Recall@10 Recall@50 Recall@100

Baselines: BM25 LateOn Qwen3-Embedding-0.6B Qwen3-Embedding-4B Gemini-Embedding-2 GPT-5.2 Query Rewriter GPT-5.2 Multi-Hop Agent

Builds on: BRIGHT BEIR BrowseComp-Plus ColBERT PLAID

Closest papers by meaning

• Retriever-Free Retrieval-Augmented Reasoning via Corpus-Traversing MCTS (NeurIPS 2026 · similarity 0.87)
• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval (ICLR 2025 · similarity 0.86)
• Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation (NeurIPS 2026 · similarity 0.85)
• Retrieval is Not Enough: Enhancing RAG through Test-Time Critique and Optimization (NeurIPS 2025 · similarity 0.85)
• Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievals (NeurIPS 2025 · similarity 0.84)
• Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction (NeurIPS 2026 · similarity 0.84)