Research map / Large language models / Retrieval and knowledge
InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem
Shuofei Qiao, Yunxiang Wei, Xuehai Wang, Bin Wu, Boyang XUE, Ningyu Zhang, Hossein A. Rahmani, Wang Yanshan, Qiang Zhang, Keyan Ding, Jeff Z. Pan, Huajun Chen and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use full InnoEval pipeline instead of strongest baseline ScholarEval | DeepSeek-V3.2 · point-wise 3-class | macro F1 (%): 58.38 -> 74.56 | Improved |
| use InnoEval instead of ScholarEval | DeepSeek-V3.2 · group-wise best selection | accuracy (%): 49.42 -> 65.12 | Improved |
| use InnoEval instead of ScholarEval | DeepSeek-V3.2 · pair-wise hard | accuracy (%): 60.00 -> 63.00 | Improved |
| add grounding agent that filters retrieved reports (ablation) | InnoEval · point/pair/group-wise | accuracy and F1 | Improved |
| add personalized reviewer personas instead of plain LLM-as-judge (ablation) | InnoEval · point-wise and group-wise | accuracy and F1 | Improved |
| add web and code search beyond literature only (ablation) | InnoEval · pair-wise and group-wise | accuracy | Improved |
| swap backbone DeepSeek-V3.2 for o4-mini (ablation) | InnoEval · all tasks | accuracy and F1 | Mixed |
| add diverse academic personas vs vanilla test-time scaling (ablation) | InnoEval · point-wise and group-wise | F1 / accuracy | Improved |
| use a single persona instead of no persona (ablation) | InnoEval · group-wise ranking | accuracy | Hurt |
| use InnoEval feedback in ResearchAgent idea iteration | ResearchAgent · idea generation quality | average score | Improved |
| use ScholarEval feedback in ResearchAgent idea iteration | ResearchAgent · idea generation quality | average score | Hurt |
| use ScholarEval search engine instead of InnoEval's | evaluation · search quality | diversity | Hurt |
Closest papers by meaning
- GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation (ICLR 2025 · similarity 0.85)
- SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks (NeurIPS 2025 · similarity 0.85)
- Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers (ICLR 2025 · similarity 0.84)
- MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome (NeurIPS 2026 · similarity 0.84)
- InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.83)
- The Alien Space of Science: Sampling Coherent but Cognitively Unavailable Research Directions (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
