Research map / Graph learning / GNN architectures
Dynamic Rescaling for Training GNNs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that dynamic rescaling of GAT (and GCN) parameters and gradients, which leaves the loss invariant, can steer GNN training toward balanced relative gradients or controlled layer-wise learning order, with effects that differ between heterophilic and homophilic graphs.
Problem
Understanding and controlling GNN training dynamics; existing balanced-norm work only balances at initialization, and layer learning order and learning-rate interactions in GNNs are poorly understood.
Approach
Extends rescale-invariance-based balancing from initialization to during training by dynamically rescaling GAT neurons under any criterion of parameters and gradients, and proposes balancing by relative gradients (gradient divided by parameter) together with weight norms.
Key results
- Relative-gradient balancing (DRRG/DRC) improves over no dynamic rescaling on roman-empire, e.g., DRC 0.6731 vs 0.4978 accuracy for a 5-layer GAT.
- Concentrating initial learning on the first layer speeds convergence and improves generalization on the homophilic synthetic task, while heterophilic real-world tasks benefit more from training in balance.
- Rebalancing during training with larger learning rates (0.01) yields better generalization than either component alone, e.g., 95.28 test accuracy on synthetic task with DR at lr 0.01 vs 94.08 at lr 0.001.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use relative-gradient dynamic rescaling instead of no rescaling | 5-layer GAT · roman-empire | accuracy: .4978 ± .0209 -> .5422 ± .0234 | Helped |
| Use relative-gradient dynamic rescaling instead of no rescaling | 5-layer GAT · amazon-ratings | accuracy: .4545 ± .0043 -> .4540 ± .0029 | No clear effect |
| Use relative-gradient dynamic rescaling instead of no rescaling | 5-layer GAT · tolokers | ROC AUC: .6493 ± .008 -> .6637±.0088 | Helped |
| Use relative-gradient dynamic rescaling instead of no rescaling | 5-layer GAT · questions | ROC AUC: .5829±.0172 -> .5869 ± .0145 | No clear effect |
| Use relative-gradient dynamic rescaling instead of no rescaling | 5-layer GAT · minesweeper | ROC AUC: .5057 ± .0058 -> .5065 ± .0076 | No clear effect |
| Combine weight-norm and relative-gradient rescaling | 5-layer GAT · roman-empire | accuracy: .4978 ± .0209 -> .6731 ± .0149∗ | Helped |
| Combine weight-norm and relative-gradient rescaling | 5-layer GAT · amazon-ratings | accuracy: .4545 ± .0043 -> .4526 ± .0042 | No clear effect |
| Combine weight-norm and relative-gradient rescaling | 5-layer GAT · tolokers | ROC AUC: .6493 ± .008 -> .6642 ± 0.0064 | Helped |
| Combine weight-norm and relative-gradient rescaling | 5-layer GAT · questions | ROC AUC: .5829±.0172 -> .5696±.0132 | Hurt |
| Combine weight-norm and relative-gradient rescaling | 5-layer GAT · minesweeper | ROC AUC: .5057 ± .0058 -> .5080 ± 0.0053 | Helped |
| Add gradient clipping without dynamic rescaling (ablation) | 5-layer GAT · roman-empire | accuracy: 0.4978 ± 0.0209 -> 0.3711 ± 0.0357 | Hurt |
| Add gradient clipping without dynamic rescaling (ablation) | 5-layer GAT · amazon-ratings | accuracy: 0.4545 ± 0.0043 -> 0.4535 ± 0.0025 | No clear effect |
Surprising findings
- Balancing with respect to weight norms was less effective than balancing with respect to relative gradients, mirroring CNN findings.
- Grokking-like behavior can be induced by delaying learning of middle or last hidden layers, not only the last layer, and in one real-world case it improved generalization.
Limitations
- Dynamic rescaling requires a rescale-invariant architecture; the Tanh experiment shows effects are noisier without invariance.
- Numerical instability from direct parameter/gradient manipulation requires gradient clipping.
- Frequent rebalancing adds computational overhead; experiments are limited and the paper is not state-of-the-art focused.
Experimental setup
Datasets: roman-empire amazon-ratings tolokers questions minesweeper synthetic node classification (Erdos-Renyi graph)
Benchmarks: roman-empire amazon-ratings tolokers questions minesweeper
Metrics: accuracy ROC AUC convergence epoch
Baselines: Standard training without dynamic rescaling DRW (dynamic rescaling by weight norms)
Builds on: Balanced initialization for GATs (Rahman et al.) Rescale invariance / conservation laws (Neyshabur et al.) Grokking Gradient flow with finite learning rates (Ji et al.)
Closest papers by meaning
- How Graph Neural Networks Learn: Lessons from Training Dynamics (ICML 2024 · similarity 0.90)
- On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learning (NeurIPS 2025 · similarity 0.89)
- Locality-Aware Graph Rewiring in GNNs (ICLR 2024 · similarity 0.89)
- On the Interplay between Graph Structure and Learning Algorithms in Graph Neural Networks (ICML 2025 · similarity 0.88)
- Learning to Reweight for Generalizable Graph Neural Network (AAAI 2024 · similarity 0.88)
- Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.