Research map / Large language models / Agents, code and math
SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models
Thinh Pham, Nguyen Phan Nguyen, Pratibha Zunjare, Weiyuan Chen, Yu-Min Tseng, Tu Vu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add FreshPrompt search (ablation) | DeepSeek-R1-671B · Seal-Hard | accuracy (%): 22.4 -> 11.0 | Hurt |
| add FreshPrompt search (ablation) | DeepSeek-R1-671B · Seal-Hard, never-changing questions | accuracy (%): -17.7% | Hurt |
| add FreshPrompt search (ablation) | DeepSeek-R1-671B · Seal-Hard, 2025 questions | accuracy (%): +5.4% | Improved |
| add ChatGPT built-in search | GPT-4.1 · Seal-Hard | accuracy (%): +5.5% | Improved |
| add FreshPrompt search instead of built-in search | GPT-4.1 · Seal-Hard | accuracy (%): 15.0 -> 14.6 | No clear effect |
| add built-in search | GPT-4.1-mini · Seal-Hard | accuracy (%): 13.8 -> 11.8 | Hurt |
| add FreshPrompt search | Qwen3-235B-A22B · Seal-Hard | accuracy (%): +7.1% | Improved |
| add FreshPrompt search | DeepSeek-R1-Distill-Qwen-14B · Seal-Hard | accuracy (%): +9.7% | Improved |
| increase reasoning effort from low to high (ablation) | o3 · Seal-0 | accuracy (%): 11.7 -> 17.1 -> 14.4 | Mixed |
| increase reasoning effort from low to high (ablation) | o4-mini · Seal-0 | accuracy (%): 6.3 -> 5.4 -> 4.5 | Hurt |
| increase reasoning effort from low to high (ablation) | o3-mini · Seal-0 | accuracy (%): 1.8 -> 2.7 -> 1.8 | No clear effect |
| provide only the gold document (no hard negatives) (ablation) | GPT-4.1 · LongSeal | accuracy (%): 48.0 (vs 15.0 w/o search) | Improved |
Closest papers by meaning
- Making Retrieval-Augmented Language Models Robust to Irrelevant Context (ICLR 2024 · similarity 0.88)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.87)
- Long-form factuality in large language models (NeurIPS 2024 · similarity 0.87)
- BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack (NeurIPS 2024 · similarity 0.87)
- ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models (NeurIPS 2026 · similarity 0.87)
- AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
