Research map / Large language models / Agents, code and math
ProSearch: Benchmarking Multi-Constraint Protocol Retrieval in Experimental Science
Wenliang Liang, Zhiyuan Ning, Haolin Chen, Yuanchun Zhou, Wenjuan Cui, Yi Du
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ProSearch, a benchmark for evaluating retrievers on multi-constraint experimental protocol retrieval in science.
Problem
Retrieval systems supporting experimental design and method reproduction must find protocols satisfying interdependent constraints (materials, operating conditions), but systematic evaluation of retrievers in such multi-constraint settings is lacking.
Approach
A benchmark dedicated to multi-constraint protocol retrieval in experimental science, providing a systematic evaluation setting for retrievers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| rerank top-100 with a cross-encoder or LLM reranker (Qwen3-Reranker) | Qwen3-Embedding-8B · ProSearch avg | nDCG@10: 0.290 -> 0.474 | Improved |
| add DIVER-QExpand query expansion | Qwen3-Embedding-8B · ProSearch avg | nDCG@10: 0.290 -> 0.317 (+0.027) | Improved |
| add reasoning-augmented query rewriting | Qwen3-Embedding-8B · ProSearch avg | nDCG@10: 0.290 -> 0.310 (+0.020) | Improved |
| decompose the query into sub-queries | Qwen3-Embedding-8B · ProSearch avg | nDCG@10: 0.290 -> 0.232 | Hurt |
| use HyDE hypothetical documents as the query | Qwen3-Embedding-8B · ProSearch avg | nDCG@10: 0.290 -> 0.272 | Hurt |
| rewrite query as structured query | Qwen3-Embedding-8B · ProSearch avg | nDCG@10: 0.290 -> 0.246 | Hurt |
| TongSearch-QR query rewriting | Qwen3-Embedding-8B · ProSearch avg | nDCG@10: 0.290 -> 0.285 | No clear effect |
| use scientific-domain pretrained retrievers instead of general dense retrievers | dense retrievers · ProSearch avg | nDCG@10: best scientific 0.158 vs best general 0.290 | Hurt |
| use reasoning-oriented retrievers instead of general dense retrievers | dense retrievers · ProSearch avg | nDCG@10: best reasoning 0.255 vs 0.290 | No clear effect |
| expand the corpus from method paragraphs to all body paragraphs (ablation) | Qwen3-Embedding-8B · ProSearch avg | nDCG@10: 0.290 -> 0.206 | Hurt |
| add explicit structured constraint matching over extracted records (ablation) | Qwen3-Embedding-8B · ProSearch avg | nDCG@10: 0.290 -> 0.774 | Improved |
| break structured-match ties with dense similarity (ablation) | Structured Diagnostic · ProSearch avg | nDCG@10: 0.774 -> 0.814 | Improved |
Experimental setup
Benchmarks: ProSearch
Closest papers by meaning
- KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval (ICLR 2024 · similarity 0.85)
- SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.85)
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.85)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.84)
- CURIE: Evaluating LLMs on Multitask Scientific Long-Context Understanding and Reasoning (ICLR 2025 · similarity 0.84)
- Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
