Research map / Graph learning / Graph self-supervised learning
Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TAPE, which prompts an LLM (GPT-3.5, Llama2) for ranked predictions and explanations on text-attributed graph nodes, fine-tunes a small LM (DeBERTa) to turn the explanations into node features, and trains GNNs on the averaged predictions from these features.
Problem
Node classification on text-attributed graphs (TAGs) where shallow text features are weak and LM-based pipelines are costly, and where LLMs accessed only via text APIs (LMaaS) cannot be fine-tuned or expose logits.
Approach
Use LLM-generated textual explanations and ranked predictions as an intermediate medium: a small LM "interpreter" fine-tuned on those explanations converts them into frozen node features for any downstream GNN, avoiding LM-GNN iterative training and requiring only text-level LLM access.
Key results
- On ogbn-arxiv, TAPE with RevGAT reaches 77.50% test accuracy, the best reported, versus 76.57% for GLEM-G-Step style baselines' top figure in the table (GLEM-G-Step 76.57%).
- TAPE training on ogbn-arxiv takes 3.76 min for the GNN step versus 9.18h for GLEM, reported as a 2.88x speedup over GLEM.
- On tape-arxiv23 (post-GPT-3.5 cutoff), TAPE with RevGAT reaches 84.23% versus 73.56% for zero-shot GPT-3.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add explanation and prediction features alongside original-text features (ablation) | RevGAT on ogbn-arxiv · ogbn-arxiv test | accuracy: 0.7504 ± 0.0020 -> 0.7750 ± 0.0012 | Helped |
| Add original-text and prediction features alongside explanation features (ablation) | RevGAT on ogbn-arxiv · ogbn-arxiv test | accuracy: 0.7529 ± 0.0052 -> 0.7750 ± 0.0012 | Helped |
| Add original-text and explanation features alongside prediction features (ablation) | RevGAT on ogbn-arxiv · ogbn-arxiv test | accuracy: 0.7519 ± 0.0031 -> 0.7750 ± 0.0012 | Helped |
| Use Llama2-13b-chat instead of GPT-3.5 to generate predictions and explanations | TAPE with RevGAT · ogbn-arxiv | accuracy: 0.7750 ± 0.0012 -> 0.7619 ± 0.0027 | Hurt |
| Move the paper title before the abstract in the prompt (ablation) | GPT-3.5 zero-shot classification · ogbn-arxiv | accuracy: 0.720 -> 0.695 | Hurt |
| Instruct the LLM to focus only on content in the actual text (ablation) | GPT-3.5 zero-shot classification · ogbn-arxiv | accuracy: 0.720 -> 0.695 | Hurt |
| Request step-by-step categorization reasoning in the prompt (ablation) | GPT-3.5 zero-shot classification · ogbn-arxiv | accuracy: 0.720 -> 0.705 | Hurt |
| Use TAPE's decoupled training instead of GLEM's iterative training | DeBERTa-base + RevGAT on ogbn-arxiv · ogbn-arxiv | total training time: 9.18h -> 3.76min; 2.88× speedup | Helped |
Surprising findings
- Llama-2-13b-chat zero-shot explanations give 76.19% on ogbn-arxiv when used through TAPE, close to GPT-3.5's 77.50%, despite much lower zero-shot accuracy (44.23%).
Limitations
- Requires manually crafted, dataset-specific prompts whose quality affects performance.
- LLM querying cost and time for large graphs (about $128 and about 9 hours for ogbn-arxiv via API).
- ogbn-products evaluated only on a sampled subgraph due to compute budget.
Experimental setup
Datasets: Cora PubMed ogbn-arxiv ogbn-products tape-arxiv23
Benchmarks: Cora PubMed ogbn-arxiv ogbn-products tape-arxiv23
Metrics: accuracy
Baselines: GCN GraphSAGE RevGAT MLP GIANT LM finetune (DeBERTa) GLEM zero-shot GPT-3.5
Builds on: GLEM DeBERTa RevGAT GIANT
Closest papers by meaning
- Generalization Principles for Inference over Text-Attributed Graphs with Large Language Models (ICML 2025 · similarity 0.91)
- Label-free Node Classification on Graphs with Large Language Models (LLMs) (ICLR 2024 · similarity 0.91)
- Leveraging Large Language Models for Node Generation in Few-Shot Learning on Text-Attributed Graphs (AAAI 2025 · similarity 0.90)
- SSTAG: Structure-Aware Self-Supervised Learning Method for Text-Attributed Graphs (NeurIPS 2025 · similarity 0.89)
- LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification (ICML 2025 · similarity 0.89)
- TGLsta: Low-resource Textual Graph Learning with Semantic and Topological Awareness via LLMs (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.