Research map / Graph learning / Graph self-supervised learning
Leveraging Contrastive Learning for Enhanced Node Representations in Tokenized Graph Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GCFormer, a tokenized graph Transformer that builds both positive (top-k similar) and negative (sampled dissimilar) node token sequences in attribute and topology feature spaces, fuses them with signed attention, and adds a contrastive loss for node classification.
Problem
Tokenized graph Transformers build node token sequences only from a small set of high-similarity nodes, discarding information from dissimilar nodes that could help learn distinguishable node representations for node classification.
Approach
A hybrid token generator produces positive tokens (top-k similar nodes) and negative tokens (uniformly sampled from the rest) in both attribute and topology feature spaces; a Transformer backbone processes each sequence separately with a virtual token, fuses target and virtual outputs via signed subtraction, and a contrastive loss pulls the target toward positive tokens and pushes it from negative
Key results
- GCFormer achieves the best accuracy on all eight datasets in Table 2, e.g., 96.03% on BlogCatalog vs. 94.42% for NAGphormer and 92.09% on Computer vs. 90.79%.
- Against GCL-based methods (ClusterSCL, CoCoS, NCLA), GCFormer also reports the highest accuracy on every dataset in the appendix comparison.
- Ablation shows removing negative tokens and contrastive loss (GCFormer-N) degrades results, with GCFormer-C beating -N on over half of datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GCFormer instead of the strongest reported baseline | Node classification · Photo | accuracy: 95.47 -> 95.65 | Helped |
| Use GCFormer instead of the strongest reported baseline | Node classification · ACM | accuracy: 93.92 -> 94.32 | Helped |
| Use GCFormer instead of the strongest reported baseline | Node classification · Computer | accuracy: 91.31 -> 92.09 | Helped |
| Use GCFormer instead of the strongest reported baseline | Node classification · Corafull | accuracy: 69.34 -> 69.53 | Helped |
| Use GCFormer instead of the strongest reported baseline | Node classification · BlogCatalog | accuracy: 94.77 -> 96.03 | Helped |
| Use GCFormer instead of the strongest reported baseline | Node classification · UAI2010 | accuracy: 76.94 -> 77.57 | Helped |
| Use GCFormer instead of the strongest reported baseline | Node classification · Flickr | accuracy: 86.85 -> 87.90 | Helped |
| Use GCFormer instead of the strongest reported baseline | Node classification · Romanempire | accuracy: 74.94 -> 75.38 | Helped |
| Add negative token sequences while omitting contrastive loss (ablation) | GCFormer-N · Most datasets; three citation networks differ | accuracy | Mixed |
| Add contrastive learning loss (ablation) | GCFormer · All datasets | accuracy | Helped |
| Pass negative tokens through Transformer layers before contrastive learning (ablation) | GCFormer · All datasets | accuracy | Helped |
| Use negative-token representations in signed aggregation as well as contrastive learning (ablation) | GCFormer · All datasets | accuracy | Helped |
Surprising findings
- Ablation shows the contrastive-only variant (-C) underperforms the no-negative variant (-N) on the three citation networks, indicating negative tokens do not always help.
Limitations
- Unified sampling strategy across graph types; performance is sensitive to token sampling sizes p_k and n_k.
- Gains from negative tokens vary by graph type (e.g., citation networks).
- Evaluated only on node classification with a fixed 50/25/25 random split.
Experimental setup
Datasets: Photo ACM Computer Corafull BlogCatalog UAI2010 Flickr Romanempire
Benchmarks: Photo ACM Computer Corafull BlogCatalog UAI2010 Flickr Romanempire
Metrics: accuracy (mean ± stdev)
Baselines: APPNP SGC GPRGNN FAGCN ACM-GCN SGFormer ANS-GT Specformer VCR-Graphormer GraphGPS NAGphormer ClusterSCL
Builds on: NAGphormer signed attention graph contrastive learning
Closest papers by meaning
- GNN-Transformer Cooperative Architecture for Trustworthy Graph Contrastive Learning (AAAI 2025 · similarity 0.89)
- ReGCL: Rethinking Message Passing in Graph Contrastive Learning (AAAI 2024 · similarity 0.89)
- Tokenphormer: Structure-aware Multi-token Graph Transformer for Node Classification (AAAI 2025 · similarity 0.89)
- Graph Structure Refinement with Energy-based Contrastive Learning (AAAI 2025 · similarity 0.89)
- Rethinking Tokenized Graph Transformers for Node Classification (NeurIPS 2025 · similarity 0.89)
- Does GCL Need a Large Number of Negative Samples? Enhancing Graph Contrastive Learning with Effective and Efficient Negative Sampling (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.