Research map / Large language models / Language, safety and interpretability
Eliminating Position Bias of Language Models: A Mechanistic Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PINE, a training-free zero-shot inference method that makes inter-document attention bidirectional and re-assigns document positions by attention-based importance to eliminate position bias in LLM outputs.
Problem
Language models favor content based on its position in context (position bias), hurting LM-as-a-judge, retrieval-augmented QA, and other tasks where the relative order of documents should not matter.
Approach
Attributes position bias to causal attention and RoPE; makes inter-document attention bidirectional (intra-document stays causal) and reassigns document positions by importance scores computed from position-free attention, yielding inter-document position-invariant inference with a proof.
Key results
- On RewardBench reasoning, PINE with Llama-3-70B-Instruct reaches 87.6% vs 78.9% vanilla shuffled, surpassing GPT-4-0125-preview (86.9%) and GPT-4o (86.6%) per the paper.
- PINE improves Llama-3-8B-Instruct reasoning subset from 65.3% (shuffled vanilla) to 73.4%, and Qwen1.5-7B-Chat full set from 60.9% to 61.5%.
- PINE improves Qwen1.5-7B math reasoning accuracy on R-GSM by 12.6% and 110B by 5.3% relative to vanilla inference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply PINE instead of vanilla inference | Llama-3-8B-Instruct · RewardBench Reasoning set | accuracy: 65.3 -> 73.4 (+8.1) | Helped |
| apply PINE instead of vanilla inference | Llama-3-70B-Instruct · RewardBench Reasoning set | accuracy: 78.9 -> 87.6 (+8.7) | Helped |
| apply PINE instead of vanilla inference | Qwen-1.5 models · RewardBench Reasoning set | accuracy: +11.7, +6.9, +3.7, +9.9, -1.1, +0.8, +5.8, +8.2 | Helped |
| apply PINE instead of vanilla inference | Qwen-1.5-72B-Chat · RewardBench Full set | accuracy: 72.8 -> 71.8 (-1.1) | Hurt |
| use PINE instead of the strongest compared baseline | Llama-3-8B-Instruct · RewardBench Reasoning set | accuracy: 56.5 -> 73.4 | Helped |
| use PINE instead of the strongest compared baseline | Qwen1.5-7B-Chat · RewardBench Reasoning set | accuracy: 53.4 -> 63.0 | Helped |
| apply PINE instead of vanilla inference | Llama-3-70B-Instruct · Retrieval-augmented QA, 10 documents | performance: +1.2 | Helped |
| apply PINE instead of vanilla inference | Llama-3-70B-Instruct · Retrieval-augmented QA, 20 documents | performance: -2.0 | Hurt |
| add position reassignment by importance scores (ablation) | PINE without position reassignment · Retrieval-augmented QA | performance: +0.3; +1.5 with 20 documents | Helped |
| assign positions in reverse importance order (ablation) | PINE · Retrieval-augmented QA | performance | Hurt |
| add PINE to property-conditioned molecule generation | Llama · QM9 | property criteria | Mixed |
| add PINE to condition-based math reasoning | Qwen1.5-7B and Qwen1.5-110B · R-GSM | accuracy: 12.6%; 5.3% | Helped |
Surprising findings
- Qwen1.5-72B-Chat shows no gain (-1.1 average) with PINE, and PINE underperforms vanilla on the 20-document RAG setting by 2.0 points.
- Summing token importance caused models to shift from position bias to length bias, so averaging was used.
Limitations
- PINE incurs extra inference computation (about 2x to 8x wall time) and its implementation is not optimized.
- Gains are inconsistent: Qwen1.5-72B-Chat shows a drop and RAG with 20 documents is slightly worse than vanilla.
- Elimination assumes documents are position-agnostic; it does not apply where order carries meaning.
Experimental setup
Datasets: RewardBench Lost in the Middle retrieval-augmented QA data QM9 R-GSM
Benchmarks: RewardBench R-GSM QM9
Metrics: accuracy MAE
Baselines: Vanilla inference NIA PCW Structured Prompting permutation calibration
Builds on: Parallel Context Windows Lost in the Middle RoPE RewardBench
Closest papers by meaning
- Rethinking Addressing in Language Models via Contextualized Equivariant Positional Encoding (ICML 2025 · similarity 0.88)
- Identifying and Mitigating Position Bias of Multi-image Vision-Language Models (CVPR 2025 · similarity 0.87)
- GOLD PANNING: Strategic Context Shuffling for Needle-in-Haystack Reasoning (NeurIPS 2026 · similarity 0.87)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.87)
- Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs (NeurIPS 2024 · similarity 0.86)
- UniBias: Unveiling and Mitigating LLM Bias through Internal Attention and FFN Manipulation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.