Research map / Large language models / AI and society
ResearchTown: Simulator of Human Research Community
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ResearchTown, a multi-agent LLM simulator that models a research community as an agent-data graph with TextGNN message passing, evaluated on the ResearchBench node-masking benchmark.
Problem
Whether LLM multi-agent systems can simulate collaborative human research activities such as paper reading, paper writing, and review writing, and how to evaluate such simulations objectively.
Approach
Represent the research community as an agent-data graph (researchers as agent nodes, papers as data nodes) and define LLM-based text-space message passing (TextGNN) so that paper reading, writing, and reviewing are instantiations of GNN layers; evaluate by masking paper/review nodes and measuring embedding similarity to ground truth.
Key results
- Global aggregation (ResearchTown) achieves overall paper-writing similarity of 67.51 vs 65.30 for AGG-data and 46.08 for AGG-self (text-embedding-3-large).
- Review writing strength/weakness similarity around 51 and 47, with review score difference of 1.55 for AGG-global versus 1.19 for AGG-agent.
- Human-LLM similarity judgments correlate with Pearson 0.61; simulated papers are slightly less novel than real ones (5.50 vs 5.90 human scores).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add both agent and data neighborhood information instead of data alone | ResearchTown paper writing · ResearchBench paper writing, Overall | similarity: 65.30 -> 67.51 | Helped |
| Add neighboring researchers to data aggregation | ResearchTown paper writing · ResearchBench paper writing, Hard | similarity: 56.02 -> 60.89 | Helped |
| Use all neighboring agents and data instead of only the target node | ResearchTown review writing · ResearchBench review writing | review score difference (ΔS): 1.27 -> 1.55 | Hurt |
| Use related data nodes without agents instead of only the target node | ResearchTown review writing · ResearchBench review writing | weakness similarity: 47.16 -> 47.62 | Helped |
| Increase paper-writing agents from 1 to 2 (ablation) | ResearchTown with agent aggregation · Paper writing | similarity: 49.0 -> 52.7 | Helped |
| Increase reviewers from 1 to 5 (ablation) | ResearchTown review writing · Review writing | strength similarity: 50.8 -> 51.5 | Helped |
| Select papers cited in related work rather than using all papers (ablation) | ResearchTown paper writing · Paper writing | similarity: 66.4 -> 66.7 | Helped |
| Use only papers cited in the introduction (ablation) | ResearchTown paper writing · Paper writing | similarity: 66.4 -> 65.2 | Hurt |
| Include papers cited in other sections (ablation) | ResearchTown paper writing · Paper writing | similarity: 66.4 -> 58.4 | Hurt |
| Use global aggregation instead of data aggregation (ablation) | ResearchTown paper writing · ResearchBench paper writing, Overall | similarity: 65.30 -> 67.51 | Helped |
| Use global aggregation instead of self aggregation (ablation) | ResearchTown review writing with GPT-4o-mini · ResearchBench review writing | review score difference (ΔS): 1.27 -> 1.51 | Hurt |
Surprising findings
- Cited papers (data aggregation) contribute more than author agents to paper-writing similarity, and the global setting only modestly improves over data-only.
- Global aggregation increased review score differences for GPT-4o-mini (stricter novelty judgments, average score dropping from 5.3 to 5.0).
Limitations
- Simulated papers and reviews are less novel and feasible than real ones and cannot be used directly; outputs become vague when combining too many domains.
- Review writing similarity is lower than paper writing, especially for weaknesses, due to noisy real review data.
- Evaluation relies on embedding similarity to ground-truth papers that are themselves imperfect; LLM judge has low correlation with humans on intrinsic quality.
Experimental setup
Datasets: ResearchBench (PaperBench, HighImpactPaperBench, ReviewBench) arXiv metadata Semantic Scholar author data OpenReview ICLR 2024 reviews
Benchmarks: ResearchBench
Metrics: embedding similarity (text-embedding-3-large, voyage-3) fine-grained LLM-judged similarity novelty feasibility review score difference Pearson correlation with human judgments
Baselines: AGG-self AGG-agent AGG-data
Builds on: graph neural networks message passing text-attributed graphs multi-agent LLM frameworks
Closest papers by meaning
- InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.87)
- ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows (ICLR 2026 · similarity 0.87)
- Deep Research Arena: The First Exam of LLMs’ Research Abilities via Seminar-Grounded Tasks (AAAI 2026 · similarity 0.87)
- AAAR-1.0: Assessing AI’s Potential to Assist Research (ICML 2025 · similarity 0.86)
- SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.86)
- CycleResearcher: Improving Automated Research via Automated Review (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.