Research map / Graph learning / Graph self-supervised learning
GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GraphEval, which decomposes research ideas into viewpoint-nodes linked into a graph, then evaluates them with training-free label propagation (GraphEval-LP) or a weighted GNN with a temporal novelty component (GraphEval-GNN).
Problem
LLM-based evaluation of research ideas is unstable, biased toward positive decisions, and misses factual errors and novelty; small fine-tuned models also struggle with complex idea semantics.
Approach
Break each idea into viewpoint nodes with a small prompted LLM, connect them via LLM relation extraction and/or BERT cosine similarity into a viewpoint-graph shared across ideas, and propagate quality labels with label propagation or a GNN; novelty is handled by adding temporal features and synthetic plagiarized negatives.
Key results
- On the ICLR Papers test set, GraphEval-GNN reaches 76% accuracy and 43.59% macro F1, versus 26.01% F1 for fine-tuned BERT and at most 10.05% for prompt-based baselines.
- On the AI Researcher dataset, GraphEval-GNN reaches 73.33% accuracy and 67.13% macro F1, compared with 53.33% F1 for fine-tuned BERT.
- GraphEval-LP (training-free) reaches 70% accuracy and 32.16% macro F1 on ICLR Papers at normed cost 0.08, versus 1.00 for the most expensive Research Agent (72B) baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GraphEval-GNN instead of Fine-tuned BERT | ICLR Papers idea evaluation · ICLR Papers | accuracy: 66.00% -> 76.00% | Helped |
| Use GraphEval-GNN instead of Fine-tuned BERT | ICLR Papers idea evaluation · ICLR Papers | macro F1: 26.01% -> 43.59% | Helped |
| Use GraphEval-GNN instead of Fine-tuned BERT | AI Researcher idea evaluation · AI Researcher | accuracy: 60.00% -> 73.33% | Helped |
| Use GraphEval-GNN instead of Fine-tuned BERT | AI Researcher idea evaluation · AI Researcher | macro F1: 53.33% -> 67.13% | Helped |
| Use GraphEval-LP instead of Fine-tuned BERT | ICLR Papers idea evaluation · ICLR Papers | accuracy: 66.00% -> 70.00% | Helped |
| Use GraphEval-LP instead of Fine-tuned BERT | ICLR Papers idea evaluation · ICLR Papers | macro F1: 26.01% -> 32.16% | Helped |
| Use GraphEval-LP instead of Fine-tuned BERT | AI Researcher idea evaluation · AI Researcher | accuracy: 60.00% -> 70.47% | Helped |
| Use GraphEval-LP instead of Fine-tuned BERT | AI Researcher idea evaluation · AI Researcher | macro F1: 53.33% -> 56.97% | Helped |
| Replace GraphEval-LP with GraphEval-GNN | GraphEval · ICLR Papers | macro F1: 32.16% -> 43.59% | Helped |
| Replace GraphEval-LP with GraphEval-GNN | GraphEval · AI Researcher | macro F1: 56.97% -> 67.13% | Helped |
| Add novelty assessment using temporal features and generated plagiarism negatives (ablation) | GraphEval-GNN · ICLR Papers plagiarized-idea detection | performance across four metrics | Helped |
Surprising findings
- Scaling the prompted LLM from 7B to 72B did not consistently improve evaluation performance; the 72B model often gave similar decisions across ideas.
- Prompt-based baselines overestimated idea quality, with very few ideas rejected.
Limitations
- Very small evaluation: 50 ICLR test papers and 66 AI Researcher papers, with 80 synthetic plagiarism cases.
- Novelty evaluation relies on artificially constructed plagiarized ideas rather than real plagiarism cases.
- Prompt-based baseline results depend on specific prompt design and label distribution hints.
Experimental setup
Datasets: ICLR Papers (2021-2023 abstracts and decisions) AI Researcher Dataset
Benchmarks: ICLR Papers (50-paper test set) AI Researcher Dataset
Metrics: accuracy macro precision macro recall macro F1 normed token cost
Baselines: Prompted LLM CoT prompt CoT-SC ToT prompt Research Agent Fine-tuned BERT
Builds on: label propagation self-consistency with chain-of-thought tree of thoughts Research Agent DistilBERT
Closest papers by meaning
- Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning (ICLR 2024 · similarity 0.85)
- Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models (NeurIPS 2024 · similarity 0.85)
- Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Data (ICML 2025 · similarity 0.85)
- GraphVis: Boosting LLMs with Visual Knowledge Graph Integration (NeurIPS 2024 · similarity 0.85)
- Large Language Model Meets Graph Neural Network in Knowledge Distillation (AAAI 2025 · similarity 0.84)
- SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.