Research map / Large language models / Alignment and preferences
Search Arena: Analyzing Search-Augmented LLMs
Mihran Miroyan, Tsung-Han Wu, Logan King, Tianle Li, Jiayi Pan, Xinyan Hu, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Trevor Darrell, Narges Norouzi, Joseph E. Gonzalez
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a reasoning model for search-augmented chat | search-augmented LLMs · Search Arena | average win rate: top three models over 60.0% average win rate | Improved |
| increase search context size from medium to high (ablation) | sonar-pro · Search Arena | average win rate: 57.6% -> 63.9% | Improved |
| increase search context size from medium to high (ablation) | GPT-4o search models · Search Arena | average win rate | No clear effect |
| longer responses | user preference (Bradley-Terry) · Search Arena | BT coefficient: beta_length = 0.334 | Improved |
| longer responses on Factual Lookup prompts | user preference (Bradley-Terry) · Search Arena (Factual Lookup) | BT coefficient: 0.334 -> 0.156 | Mixed |
| more cited sources | user preference (Bradley-Terry) · Search Arena | BT coefficient: beta_citations = 0.209 | Improved |
| cite Wikipedia | user preference (Bradley-Terry) · Search Arena | BT coefficient: beta_wiki = -0.071 | Hurt |
| cite tech, community or social platforms | user preference (Bradley-Terry) · Search Arena | BT coefficient: 0.073, 0.061, 0.057 | Improved |
| add irrelevant (non-supporting) citations | user preference (Bradley-Terry) · Search Arena (citation subset) | BT coefficient: beta_irrelevant = 0.27 vs beta_support = 0.29 | Improved |
| add contradicting claim-citation pairs | user preference (Bradley-Terry) · Search Arena (citation subset) | BT coefficient | No clear effect |
| add web search to a non-search LLM (ablation) | Gemini-2.5 Pro Experimental · Text Arena | vote preference: 143 search vs 156 non-search, p=0.244 | No clear effect |
| add web search to a non-search LLM (ablation) | Gemini-2.5 Pro Experimental · Text Arena (Factual Lookup) | vote preference: p-value=0.012 | Improved |
Closest papers by meaning
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.90)
- Long-form factuality in large language models (NeurIPS 2024 · similarity 0.88)
- WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia (NeurIPS 2024 · similarity 0.88)
- Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs (ICLR 2025 · similarity 0.88)
- MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (NeurIPS 2024 · similarity 0.87)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
