Research map / Probabilistic and scientific modeling / Treatment effects

Perplexity Trap: PLM-Based Retrievers Overrate Low Perplexity Documents

ICLR 2025 · Probabilistic and scientific modeling › Treatment effects · approach: Proximal and deep causal effect estimation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that PLM-based retrievers causally favor low-perplexity documents, explains this via gradient alignment between MLM and retrieval losses, and proposes CDC, an inference-time IV-based correction that removes the perplexity effect from relevance scores.

Problem

Source bias: PLM-based retrievers rank LLM-generated documents above semantically comparable human-written ones, but the causal mechanism behind this bias was unexplained.

Approach

Builds a causal graph with document perplexity as a treatment and document semantics as a confounder, estimates the perplexity effect with instrumental-variable 2SLS (using document source as the instrument), and theoretically links it to positive correlation between MLM and retrieval gradients; then subtracts the estimated biased effect at inference time.

Key results

• CDC with 128 estimation samples reverses or mitigates human-written bias for most retrievers without fine-tuning, with typical NDCG@3 drops under 2 points.
• Perplexity shows a significant negative causal effect on relevance scores for most retrievers (e.g., BERT -9.32 on DL19, p=1e-4), with Pearson correlations below -0.8 across temperature-varied corpora.

Measured effects reported in the paper

Surprising findings

• Stronger retrievers with higher ranking performance show more severe source bias, a trade-off the paper attributes to perplexity sensitivity.
• CDC has smaller effect on stronger retrievers such as Contriever, where the estimated perplexity effect is near zero.

Limitations

• Theory relies on strong assumptions: linear decoder, semi-orthogonal weight matrix, collinearity, and dual-encoder mean pooling.
• Evaluation covers only three datasets and six BERT-style retrievers; human evaluation is small (20 triples per dataset and temperature).
• Semantic equivalence of LLM rewrites is assumed rather than fully verified.

Experimental setup

Datasets: DL19 TREC-COVID SCIDOCS SciFact

Benchmarks: DL19 TREC-COVID SCIDOCS SciFact

Metrics: NDCG@3 Relative Delta NDCG@3

Baselines: BERT RoBERTa ANCE TAS-B Contriever coCondenser

Builds on: two-stage least squares Cocktail benchmark

Closest papers by meaning

• SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction (ICLR 2025 · similarity 0.87)
• Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievals (NeurIPS 2025 · similarity 0.86)
• Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models (ICLR 2025 · similarity 0.84)
• ClashEval: Quantifying the tug-of-war between an LLM’s internal prior and external evidence (NeurIPS 2024 · similarity 0.84)
• Contradiction Retrieval via Contrastive Learning with Sparsity (ICML 2025 · similarity 0.84)
• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval (ICLR 2025 · similarity 0.84)