Research map / Large language models / AI and society
SciNet: Evaluating AI Agents in Relation-Aware Scientific Literature Retrieval
Chenyang Shao, Fengli Xu, Yong Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use deep research agent o3-deep-research instead of SciBERT | retrieval model · SciNet ego-centric novelty | Novelty-Recall@50: 0.00% -> 1.47% | Improved |
| use deep research agent o4-mini-deep-research instead of SciBERT | retrieval model · SciNet ego-centric disruption | Disruption-Recall@50: 0.00% -> 4.71% | Improved |
| use gpt-4o-search instead of Qwen3-Embed | retrieval model · SciNet pair-wise | Cite-Acc: 11.34% -> 48.65% | Improved |
| use o3-deep-research instead of gpt-4o-search | retrieval model · SciNet pair-wise | CoMention-Acc: 69.94% -> 75.68% | Improved |
| use o3-deep-research instead of gpt-4o-search | retrieval model · SciNet pair-wise | CoMention-Paragraph: 7.50% -> 17.09% | Improved |
| use PaperQA instead of Qwen3-Embed | retrieval model · SciNet pair-wise | Cite-Acc: 11.34% -> 8.43% | Hurt |
| use gpt-4o-search instead of o3-deep-research | retrieval model · SciNet path-wise | Consistency: 63.28% -> 65.52% | Improved |
| use o3-deep-research instead of gpt-4o-search | retrieval model · SciNet path-wise | Connectivity: 10.30% -> 14.54% | Improved |
| use o3-deep-research instead of SciBERT | retrieval model · SciNet path-wise | Rationality-Human: 1.72 -> 7.06 | Improved |
| use gpt-4o-mini-search instead of gpt-4o-search | retrieval model · SciNet ego-centric novelty | Novelty-SoS: 6.440 -> 6.363 | Hurt |
| use Deep Research System instead of Base LLM for literature review | survey generator · Literature review human ranking (40 queries) | average rank: 3.00 -> 2.20 | Improved |
| use Search-enabled LLM instead of Base LLM for literature review | survey generator · Literature review human ranking (40 queries) | average rank: 3.00 -> 3.58 | Hurt |
Closest papers by meaning
- SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.89)
- Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction (NeurIPS 2026 · similarity 0.87)
- Intern-Atlas: A Methodological Evolution Graph as Research Infrastructure for AI Scientists (NeurIPS 2026 · similarity 0.86)
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.85)
- From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents (NeurIPS 2026 · similarity 0.85)
- Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
