Research map / Large language models / Retrieval and knowledge

GOLD PANNING: Strategic Context Shuffling for Needle-in-Haystack Reasoning

NeurIPS 2026 · Large language models › Retrieval and knowledge · approach: Retrieval-augmented reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GOLD PANNING, a black-box Bayesian active-search method that reorders documents so high-belief items sit in positions with high measured diagnosticity, reaching PSC-level target identification with 30-65% fewer queries on MDQA (FLenQA) tasks.

Problem

Position bias in long-context needle-in-haystack retrieval makes LLMs unreliable; existing mitigations need white-box access, and inference-time ensembling (PSC) is stateless and assumes independent errors.

Approach

Treat each context position as a noisy binary detector with calibrated TPR/FPR, maintain Bayesian log-odds beliefs over document relevance from cited IDs, and greedily assign high-belief documents to the most diagnostic positions ("signal anchoring") rather than high-entropy ones.

Key results

• On Gemma-3-12B (MonoRel), GP-Belief reaches F1 about 0.97 at iteration 4, matching PSC at iteration 8.
• On Gemma-3-12B (PIR), GP-Belief achieves F1 about 0.88 at iteration 4 versus 0.72 for GP-Entropy; PSC's final iteration-8 F1 is 0.82.
• Gemma-3-27B shows a flat diagnosticity profile and no separation between methods, serving as a negative control.

Surprising findings

• Gemma-3-27B has essentially no exploitable position bias, so strategic reordering gives no gain over random shuffling.
• Cross-calibration between MonoRel and PIR profiles degrades performance only modestly, suggesting coarse positional ordering matters more than exact TPR/FPR values.

Limitations

• Conditional-independence approximation for citations within a round is not modeled explicitly; it is absorbed into marginal rates.
• Evaluated only on two synthetic-style FLenQA benchmarks with k=1 relevant document and two model families.
• The O(log N) sample complexity proof is loose and its derivation has informal steps (constants chosen conservatively, arguments hand-wavy).

Experimental setup

Datasets: MonoRel (FLenQA) PIR (FLenQA)

Benchmarks: MonoRel PIR

Metrics: F1 Spearman rho Kendall tau

Baselines: Permutation Self-Consistency GP-Entropy

Builds on: Permutation Self-Consistency Youden's index FLenQA

Closest papers by meaning

• Eliminating Position Bias of Language Models: A Mechanistic Approach (ICLR 2025 · similarity 0.87)
• RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding (ICML 2025 · similarity 0.86)
• Retriever-Free Retrieval-Augmented Reasoning via Corpus-Traversing MCTS (NeurIPS 2026 · similarity 0.86)
• Inference Scaling for Long-Context Retrieval Augmented Generation (ICLR 2025 · similarity 0.86)
• BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack (NeurIPS 2024 · similarity 0.86)
• Retrieval is Not Enough: Enhancing RAG through Test-Time Critique and Optimization (NeurIPS 2025 · similarity 0.86)