Research map / Graph learning / Graph self-supervised learning

Generalization Principles for Inference over Text-Attributed Graphs with Large Language Models

ICML 2025 · Graph learning › Graph self-supervised learning · approach: LLMs and graph foundation models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LLM-BP, a training-free zero-shot node classifier for text-attributed graphs that combines LLM2Vec task-conditioned embeddings with belief propagation whose edge homophily parameter is estimated by an LLM.

Problem

Zero-shot node classification on text-attributed graphs is hard because LLMs have limited context length for large neighborhoods and node embeddings are misaligned with the LLM token space.

Approach

Two principles: (1) task-adaptive embeddings from LLM2Vec encoders using task- and class-conditioned prompts, avoiding token-space alignment; (2) graph aggregation via loopy belief propagation on an MRF whose edge potentials are set from an LLM-estimated homophily ratio, so no training is needed.

Key results

• Task-adaptive (class-conditioned) LLM2Vec embeddings improve zero-shot accuracy by about 8.10% over the best LM-based encoders on average, per the abstract.
• Adding BP-based graph aggregation with LLM-estimated homophily gives an additional 1.71% gain over the task-adaptive embeddings.
• LLM-BP achieves the best average rank across 11 TAG datasets in zero-shot end-to-end evaluation (avg rank 2.27 vs 5.36 for the best encoder baseline).

Measured effects reported in the paper

Surprising findings

• LLMs-with-graph-adapters (LLaGA, GraphGPT) underperform plain SBert embeddings without any graph structure, suggesting their generalization comes from the small LM encoder rather than LLM reasoning over embeddings.
• Adding class information to SBert or RoBERTa encoders may degrade performance, whereas it helps LLM2Vec-style decoder-derived encoders.

Limitations

• Homophily is a single scalar edge potential, a proof-of-concept simplification that does not model class-pair-specific edge compatibilities.
• Results depend on the LLM judge's accuracy; Mistral-7B-Instruct-v0.3 failed at homophily estimation.
• Focuses on node classification; link prediction only partially covered and not a full graph foundation model.

Experimental setup

Datasets: Cora Citeseer Pubmed History Children Sportsfit Wikics Cornell Texas Wisconsin Washington

Benchmarks: Cora Citeseer Pubmed History Children Sportsfit Wikics Cornell Texas Wisconsin Washington

Metrics: accuracy macro F1 average rank

Baselines: SBert RoBERTa text-embedding-3-large bge-en-icl (LLM2Vec) GPT-3.5-turbo GPT-4o ZeroG UniGLM DGI GraphMAE OFA GOFA

Builds on: loopy belief propagation LLM2Vec Neighborhood Aggregation (NA) ZeroG UniGLM

Closest papers by meaning

• Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning (ICLR 2024 · similarity 0.91)
• SSTAG: Structure-Aware Self-Supervised Learning Method for Text-Attributed Graphs (NeurIPS 2025 · similarity 0.91)
• Leveraging Large Language Models for Node Generation in Few-Shot Learning on Text-Attributed Graphs (AAAI 2025 · similarity 0.90)
• Talk like a Graph: Encoding Graphs for Large Language Models (ICLR 2024 · similarity 0.90)
• UniGTE: Unified Graph–Text Encoding for Zero-Shot Generalization across Graph Tasks and Domains (NeurIPS 2025 · similarity 0.90)
• Dynamic Bundling with Large Language Models for Zero-Shot Inference on Text-Attributed Graphs (NeurIPS 2025 · similarity 0.89)