Research map / Graph learning / GNN architectures

No Metric to Rule Them All: Toward Principled Evaluations of Graph-Learning Datasets

ICML 2025 · Graph learning › GNN architectures · approach: Graph structure learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces RINGS, a mode-perturbation framework that evaluates graph-learning datasets via performance separability (GNN performance under structure/feature perturbations) and task-independent mode complementarity, and applies it to 13 graph-classification datasets.

Problem

Existing graph-learning benchmarks are of uneven quality (e.g., feature-only methods can match graph methods), yet there is no principled way to evaluate dataset quality itself, since the classic setup holds the dataset fixed and varies models.

Approach

Treat graph-learning datasets as having two modes (graph structure and node features) and evaluate them via dataset ablations: perturb one mode (empty/complete/random graph or features), then measure whether models separably perform differently (performance separability via KS test with permutation) and how complementary the metric spaces of the two modes are (mode complementarity and mode diversi

Key results

• Only COLLAB, MolHIV, and NCI1 satisfy the expected performance-separability relations for structure- and feature-based tasks (Peptides close), while AIDS, DD, Proteins, Reddit-B, and Reddit-M satisfy none.
• Original DD is separably outperformed by its random-graph perturbation, Enzymes by its empty-graph perturbation, and Reddit-B by complete-features and random-graph perturbations.
• Higher mean mode complementarity is generally associated with higher mean AUROC across perturbations, with exceptions for DD and Reddit-M; feature diversity is more common than structural diversity.

Measured effects reported in the paper

Surprising findings

• The original dataset is separably outperformed by perturbed variants (e.g., random graph on DD) for many datasets, indicating misaligned structure or features.
• Reddit-B and Reddit-M have high structural and feature diversity yet low performance separability, suggesting realignment rather than deprecation.

Limitations

• Performance-separability experiments mostly target message-passing GNNs, excluding other architectures.
• The norm-based comparison assumes nodes and node features are paired, so metric distortion is not measured.
• Mode complementarity is task- and model-agnostic, so it cannot make claims about (model, dataset, task) triples.

Experimental setup

Datasets: AIDS COLLAB DD Enzymes IMDB-B IMDB-M MUTAG MolHIV NCI1 Peptides Proteins Reddit-B

Benchmarks: AIDS COLLAB DD Enzymes IMDB-B IMDB-M MUTAG MolHIV NCI1 Peptides Proteins Reddit-B

Metrics: accuracy AUROC KS statistic (permutation test) mode complementarity mode diversity

Baselines: GAT GCN GIN

Builds on: graph neural networks Kolmogorov-Smirnov test diffusion distance

Closest papers by meaning

• Mosaic of Modalities: A Comprehensive Benchmark for Multimodal Graph Learning (CVPR 2025 · similarity 0.89)
• Position: Graph Learning Will Lose Relevance Due To Poor Benchmarks (ICML 2025 · similarity 0.89)
• Graph Neural Networks Are More Than Filters: Revisiting and Benchmarking from A Spectral Perspective (ICLR 2025 · similarity 0.88)
• Core Knowledge Learning Framework for Graph (AAAI 2025 · similarity 0.88)
• GraphLand: Evaluating Graph Machine Learning Models on Diverse Industrial Data (NeurIPS 2025 · similarity 0.88)
• Quality Measures for Dynamic Graph Generative Models (ICLR 2025 · similarity 0.88)