Research map / Graph learning / GNN architectures
Learning Graph Quantized Tokenizers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Graph Quantized Tokenizer (GQT), which trains a GNN with RVQ and multi-task SSL (DGI, GraphMAE2, commitment) to produce discrete node tokens, then feeds semantic-edge + PPR sequences of tokens to a vanilla Transformer.
Problem
Graph Transformers need effective and memory-efficient node tokenizers that capture local structure and long-range dependencies; graph tokenization lags behind other modalities.
Approach
Learn a hierarchical discrete graph tokenizer via residual vector quantization over GNN embeddings, trained with multi-task graph SSL (DGI, GraphMAE2, commitment loss), and serialize graphs with semantic (kNN feature) edges and PPR-selected neighbor sequences, modulated by codebook embeddings, hierarchical/positional encodings and structural gating.
Key results
- GQT reaches 73.14% accuracy on ogbn-arxiv (ablation full model) and 82.46% on ogbn-products, outperforming listed GT and GNN baselines.
- On heterophilic datasets GQT achieves 97.36 ROC-AUC on Minesweeper and 78.94 on Questions, best among compared methods.
- Under GRBCD/PR-BCD adversarial attacks, GQT shows smaller accuracy drops than RQ-VAE (e.g., 10.40% vs 14.80% on ogbn-arxiv under GR-BCD).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add GraphMAE2 objective (ablation) | GQT · ogbn-arxiv | accuracy: 71.83$\pm$0.19 -> 73.14$\pm$0.16 | Helped |
| add DGI objective (ablation) | GQT · ogbn-arxiv | accuracy: 72.71$\pm$0.24 -> 73.14$\pm$0.16 | Helped |
| add RVQ quantization (ablation) | GQT · ogbn-arxiv | accuracy: 72.84$\pm$0.23 -> 73.14$\pm$0.16 | Helped |
| add aggregated codebook embeddings (ablation) | GQT · ogbn-arxiv | accuracy: 71.34$\pm$0.16 -> 73.14$\pm$0.16 | Helped |
| add positional encoding (ablation) | GQT · ogbn-arxiv | accuracy: 72.69$\pm$0.21 -> 73.14$\pm$0.16 | Helped |
| add structural gating (ablation) | GQT · ogbn-arxiv | accuracy: 73.08$\pm$0.14 -> 73.14$\pm$0.16 | No clear effect |
| add semantic edges (ablation) | GQT · ogbn-arxiv | accuracy: 72.59$\pm$0.25 -> 73.14$\pm$0.16 | Helped |
| add semantic edges (ablation) | GQT · Minesweeper | ROC-AUC: 92.83$\pm$0.35 -> 95.28$\pm$0.44 | Helped |
| add structural gating (ablation) | GQT · Minesweeper | ROC-AUC: 93.97$\pm$0.58 -> 95.28$\pm$0.44 | Helped |
| use multi-task self-supervised objectives instead of reconstruction-only RQ-VAE training | GQT tokenizer · ogbn-arxiv | accuracy: 66.05$\pm$0.48 -> 73.14$\pm$0.16 | Helped |
| use multi-task self-supervised objectives instead of reconstruction-only RQ-VAE training | GQT tokenizer · Minesweeper | ROC-AUC: 89.69$\pm$0.35 -> 95.28$\pm$0.44 | Helped |
| use GQT tokenizer rather than RQ-VAE tokenizer | tokenizer under GR-BCD attack · PubMed | accuracy drop: 20.40% -> 15.80% | Helped |
Surprising findings
- Ablation shows that removing RVQ (keeping continuous embeddings) lowers accuracy only modestly, indicating SSL objectives matter more than quantization for performance.
Limitations
- Tokenizer objectives and sequence construction rely on preprocessing (PPR, kNN semantic edges) and hyperparameters tuned per dataset.
- Some baseline numbers are copied from prior papers rather than re-run.
Experimental setup
Datasets: CoraFull CiteSeer PubMed Computer Photo CS Physics WikiCS Squirrel Chameleon Amazon-Ratings Roman-Empire
Benchmarks: Peptides-Func Peptides-Struct COCO-SP PCQM-Contact CoraFull CiteSeer PubMed Computer Photo CS Physics WikiCS
Metrics: accuracy ROC-AUC AP MAE F1 MRR
Baselines: GCN GAT GraphGPS NAGphormer Exphormer SGFormer GOAT NodeFormer RQ-VAE
Builds on: RQ-VAE Deep Graph Infomax GraphMAE2 NAGphormer GAT
Closest papers by meaning
- Generalizable Insights for Graph Transformers in Theory and Practice (NeurIPS 2025 · similarity 0.86)
- A Graph is Worth $K$ Words: Euclideanizing Graph using Pure Transformer (ICML 2024 · similarity 0.86)
- Leveraging Contrastive Learning for Enhanced Node Representations in Tokenized Graph Transformers (NeurIPS 2024 · similarity 0.86)
- GrokFormer: Graph Fourier Kolmogorov-Arnold Transformers (ICML 2025 · similarity 0.86)
- Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence (ICML 2025 · similarity 0.85)
- G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networks (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.