Research map / Large language models / Agents, code and math
Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Runs a large blind human study (49 expert idea writers, 79 reviewers, ~300 reviews) comparing expert-written NLP research ideas with those from a Claude-3.5-based LLM ideation agent, finding AI ideas rated more novel but slightly less feasible.
Problem
Whether current LLMs can generate novel, expert-level research ideas comparable to human NLP experts, and how reliable LLM-based evaluation of ideas is.
Approach
A controlled head-to-head human study with standardized idea templates, style normalization, topic-matched blind reviews, and an LLM ideation agent that over-generates ideas, deduplicates them, and ranks them with a Swiss-tournament pairwise LLM ranker (plus a human-reranked condition).
Key results
- AI ideas were rated significantly more novel than human ideas (e.g., 5.64 vs 4.84 novelty, p<0.05 in Test 1), while feasibility was not significantly different (6.34 vs 6.61).
- Out of 4000 generated seed ideas per topic, only about 200 non-duplicate unique ideas remained, showing limited idea diversity.
- The best LLM evaluator (Claude-3.5 pairwise) reached 53.3% consistency with human scores, below the 56.1% human inter-reviewer consistency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate ideas with the LLM agent rather than recruit experts to write ideas | Research idea generation · Expert idea ratings (49 ideas; Test 2) | Novelty Score: 4.86 -> 5.62 | Helped |
| Generate ideas with the LLM agent rather than recruit experts to write ideas | Research idea generation · Expert idea ratings (49 ideas; Test 2) | Excitement Score: 4.56 -> 5.18 | No clear effect |
| Generate ideas with the LLM agent rather than recruit experts to write ideas | Research idea generation · Expert idea ratings (49 ideas; Test 2) | Feasibility Score: 6.53 -> 6.30 | No clear effect |
| Generate ideas with the LLM agent rather than recruit experts to write ideas | Research idea generation · Expert idea ratings (49 ideas; Test 2) | Expected Effectiveness Score: 5.10 -> 5.48 | No clear effect |
| Generate ideas with the LLM agent rather than recruit experts to write ideas | Research idea generation · Expert idea ratings (49 ideas; Test 2) | Overall Score: 4.69 -> 4.83 | No clear effect |
| Generate AI ideas and manually rerank them instead of using the agent's LLM ranker | AI-generated research ideas · Expert idea ratings (49 ideas; Test 2) | Novelty Score: 4.86 -> 5.78 | Helped |
| Generate AI ideas and manually rerank them instead of using the agent's LLM ranker | AI-generated research ideas · Expert idea ratings (49 ideas; Test 2) | Excitement Score: 4.56 -> 5.45 | Helped |
| Generate AI ideas and manually rerank them instead of using the agent's LLM ranker | AI-generated research ideas · Expert idea ratings (49 ideas; Test 2) | Feasibility Score: 6.53 -> 6.41 | No clear effect |
| Generate AI ideas and manually rerank them instead of using the agent's LLM ranker | AI-generated research ideas · Expert idea ratings (49 ideas; Test 2) | Expected Effectiveness Score: 5.10 -> 5.57 | No clear effect |
| Generate AI ideas and manually rerank them instead of using the agent's LLM ranker | AI-generated research ideas · Expert idea ratings (49 ideas; Test 2) | Overall Score: 4.69 -> 5.32 | No clear effect |
| Use Claude-3.5-Sonnet for pairwise paper ranking rather than GPT-4o | Accepted vs rejected ICLR 2024 LLM-related submissions · ICLR 2024 paper pair comparisons | Accuracy: 61.1% vs 71.4% | Helped |
| Use an LLM pairwise ranker to evaluate ideas rather than expert reviewers | Idea ranking · Idea ranking consistency with expert ratings | Accuracy: 56.1 vs 53.3 | Hurt |
Surprising findings
- Expert-written ideas were rated more feasible than AI ideas but the difference was not significant, while AI ideas were more novel.
- LLM evaluators agreed with human reviewers less than human reviewers agreed with each other, and an AI Scientist reviewer scored 43.3%.
Limitations
- Scope limited to prompting-based NLP research ideas, which may not generalize to other fields.
- Evaluation covers only ideas, not executed experimental outcomes; feasibility and effectiveness judgments were not validated by execution.
- Human ideas were written within a 10-day window and may not represent experts' best ideas.
Experimental setup
Datasets: 1200 ICLR 2024 LLM-related submissions (reviews and decisions, used for ranker validation) Semantic Scholar paper metadata for retrieval
Benchmarks: blind expert review of 49 human ideas vs 49 AI ideas vs 49 AI+human-rerank ideas
Metrics: novelty score excitement score feasibility score expected effectiveness score overall score inter-reviewer consistency balanced accuracy of LLM evaluators
Baselines: Human Ideas (expert-written) GPT-4o Claude-3-Opus AI Scientist reviewer
Builds on: retrieval-augmented generation repeated sampling / inference-time scaling Sentence-Transformers
Closest papers by meaning
- Language Modeling by Language Models (NeurIPS 2025 · similarity 0.87)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.86)
- AAAR-1.0: Assessing AI’s Potential to Assist Research (ICML 2025 · similarity 0.86)
- SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines (NeurIPS 2025 · similarity 0.86)
- CycleResearcher: Improving Automated Research via Automated Review (ICLR 2025 · similarity 0.86)
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.