Research map / Multi-agent systems and games / LLM agents

Agent Reviewers: Domain-specific Multimodal Agents with Shared Memory for Paper Review

ICML 2025 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds Agent Reviewers, a multi-agent LLM peer-review system with a meta-reviewer, a thumbnail-based multimodal reviewer, domain-specific reviewers backed by a shared memory pool, and an area chair, plus the Reviews-STD dataset.

Problem

Peer review feedback is scarce for many manuscripts, and existing LLM review systems produce homogenized comments, rely on shallow domain knowledge, and ignore visual elements of papers.

Approach

A training-free multi-agent review pipeline where a meta-reviewer extracts keywords to initialize domain-specific reviewers from a RAG-based shared memory pool of historical reviewed papers, a multimodal reviewer critiques a 3x3 thumbnail for layout, and reviewers discuss and revise before an area chair agent makes the final decision and updates the memory.

Key results

• Agent Reviewers reaches decision F1 0.605, MCC 0.385, Bal. Acc 0.705 with GPT-4o-mini, versus 0.410 F1 for the best baseline AgentReview (Top-3).
• Strengths-weaknesses total Jaccard improves to 0.307 from 0.245-0.285 for baselines, and total F1 to 0.453-0.462 range, with a 10.5% Jaccard gain over the best prior method.
• Removing the shared memory pool reduces decision metrics by about 38% relative, and removing multi-agent interaction by about 29%.

Measured effects reported in the paper

Surprising findings

• Removing the multimodal reviewer raised decision F1 to 0.568 in one ablation, while the full system was 0.605, so the visual gain is modest on decisions.
• Keyword-based retrieval of 5 papers vastly outperforms random sampling of 5 papers for decision F1 (0.605 vs 0.400).

Limitations

• Only papers from two AI conferences (ICLR, NeurIPS) populate the memory, limiting scope.
• Memory design is simple and could be improved with stronger RAG or online search.
• No fine-tuning of agents; the system is training-free and evaluated on a small 300-paper test set.

Experimental setup

Datasets: Reviewer2 SEA ICLR 2024 submissions

Benchmarks: ICLR 2024 subset (300 papers)

Metrics: F1-score Jaccard Recall MaxSim Decision F1 MCC Balanced Accuracy G-mean

Baselines: AgentReview AI Scientist LLM Review

Builds on: AgentReview AI Scientist LLM Review REVIEWER2 SEA

Closest papers by meaning

• From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review (NeurIPS 2025 · similarity 0.86)
• CycleResearcher: Improving Automated Research via Automated Review (ICLR 2025 · similarity 0.85)
• Mixture of Knowledge Minigraph Agents for Literature Review Generation (AAAI 2025 · similarity 0.84)
• ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (ICLR 2024 · similarity 0.84)
• Paper2Poster: Towards Multimodal Poster Automation from Scientific Papers (NeurIPS 2025 · similarity 0.84)
• Precise but Uncoupled: Reviewer Precision Does Not Guarantee Critique Uptake in Multi-Agent Math Reasoning (NeurIPS 2026 · similarity 0.84)