Research map / Graph learning / Graph self-supervised learning
Decoupling Weighing and Selecting for Integrating Multiple Graph Pre-training Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WAS, an instance-level multi-teacher distillation framework that decouples task selection (via momentum-updated siamese selector with Gumbel-softmax sampling) from task weighting to integrate multiple graph pre-training tasks for node- and graph-level downstream tasks.
Problem
Integrating multiple graph pre-training tasks: existing methods only weigh task importance via global or instance-level loss weighting and ignore task compatibility (conflicts), so performance stops improving as the task pool grows.
Approach
Treat each pre-trained task model as a teacher and combine teachers per instance via multi-teacher distillation, where a weighing module and a momentum-updated, gradient-detached siamese selecting module (with projection head and Gumbel-softmax sampling) are decoupled so that selection captures compatibility rather than just importance.
Key results
- On molecular property prediction (8 datasets, scaffold split), WAS achieves the best average rank of 1.8 among compared methods and best results on 6 of 8 datasets, e.g., BACE 80.7 and HIV 78.4 ROC-AUC.
- On node classification across 8 datasets, WAS attains average rank 1.8, beating AGSSL (3.1) and AutoSSL (4.9), e.g., Cora 84.4 and Physics 94.9.
- Ablation: removing re-weighing drops BACE from 80.7 to 65.1 ROC-AUC, and importance-based top-3 selection (75.4) underperforms random selection (77.2) on BACE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WAS to select and re-weigh multiple pre-training teachers rather than the strongest single-task baseline | GIN graph-level molecular property prediction · BACE, BBBP, SIDER, Tox21, Toxcast, MUV, HIV | ROC-AUC (%): BACE: 80.3$\pm$1.4 -> 80.7$\pm$0.5; BBBP: 70.3$\pm$1.5 -> 70.5$\pm$1.0; SIDER: 61.1$\pm$2.5 -> 60.4$\pm$0.4; Tox21: 75.2 | Helped |
| Use WAS to select and re-weigh multiple pre-training teachers rather than the strongest baseline | GCN node classification · Cora, Citeseer, CS, Physics, Photo, Computers | Accuracy: Cora: 84.2$\pm$0.3 -> 84.4$\pm$0.4; Citeseer: 73.9$\pm$0.5 -> 74.1$\pm$0.5; CS: 92.4$\pm$0.5 -> 92.6$\pm$0.3; Physics: 9 | Helped |
| Remove the selection-module MLP projection head (ablation) | WAS · BACE, BBBP, Tox21, HIV | ROC-AUC (%): BACE: 80.7$\pm$0.5 -> 77.4$\pm$1.1; BBBP: 70.5$\pm$1.0 -> 69.6$\pm$1.2; Tox21: 75.4$\pm$0.7 -> 73.9$\pm$0.8; HIV: 78.4$\ | Hurt |
| Remove re-weighing after selecting teachers (ablation) | WAS · BACE, BBBP, Tox21, HIV | ROC-AUC (%): BACE: 80.7$\pm$0.5 -> 65.1$\pm$3.7; BBBP: 70.5$\pm$1.0 -> 60.7$\pm$5.1; Tox21: 75.4$\pm$0.7 -> 64.1$\pm$3.0; HIV: 78.4$\ | Hurt |
| Use importance-based top-3 teacher selection instead of compatibility-aware selection (ablation) | WAS teacher selection · BACE, BBBP, Tox21, HIV | ROC-AUC (%): BACE: 80.7$\pm$0.5 -> 75.4$\pm$0.7; BBBP: 70.5$\pm$1.0 -> 64.4$\pm$0.9; Tox21: 75.4$\pm$0.7 -> 74.0$\pm$0.4; HIV: 78.4$\ | Hurt |
| Select all teachers rather than use WAS selection (ablation) | WAS teacher selection · BACE, BBBP, Tox21, HIV | ROC-AUC (%): BACE: 80.7$\pm$0.5 -> 75.2$\pm$1.2; BBBP: 70.5$\pm$1.0 -> 65.9$\pm$0.4; Tox21: 75.4$\pm$0.7 -> 73.6$\pm$0.9; HIV: 78.4$\ | Hurt |
| Randomly select teachers instead of using WAS selection (ablation) | WAS teacher selection · BACE, BBBP, Tox21, HIV | ROC-AUC (%): BACE: 80.7$\pm$0.5 -> 77.2$\pm$1.7; BBBP: 70.5$\pm$1.0 -> 67.2$\pm$0.5; Tox21: 75.4$\pm$0.7 -> 74.3$\pm$0.6; HIV: 78.4$\ | Hurt |
Surprising findings
- Selecting teachers purely by learned importance weight can perform worse than random selection, supporting the claim that compatibility must be modeled separately from importance.
- The teacher with the highest weight is not always the one selected, as the selector and weigher are decoupled.
Limitations
- Additional training time and parameters relative to single-task baselines, though modest.
- Selection depends on a fixed pool of classical tasks; gains from larger task pools are only shown for graph-level tasks with AGSSL extension.
Experimental setup
Datasets: GEOM BACE BBBP ClinTox SIDER Tox21 Toxcast MUV HIV Cora Citeseer Pubmed
Benchmarks: BACE BBBP ClinTox SIDER Tox21 Toxcast MUV HIV Cora Citeseer Pubmed CS
Metrics: ROC-AUC accuracy average rank
Baselines: AutoSSL ParetoGNN AGSSL AUX-TS Average-Weight Random-Select AD-GCL R-GCL GraphMVP JOAO GraphMAE SimGRACE
Builds on: AutoSSL AGSSL ParetoGNN AUX-TS multi-teacher knowledge distillation
Closest papers by meaning
- Measuring Task Similarity and Its Implication in Fine-Tuning Graph Neural Networks (AAAI 2024 · similarity 0.87)
- HeterGP: Bridging Heterogeneity in Graph Neural Networks with Multi-View Prompting (AAAI 2025 · similarity 0.85)
- Edge Prompt Tuning for Graph Neural Networks (ICLR 2025 · similarity 0.85)
- ProG: A Graph Prompt Learning Benchmark (NeurIPS 2024 · similarity 0.85)
- HG-Adapter: Improving Pre-Trained Heterogeneous Graph Neural Networks with Dual Adapters (ICLR 2025 · similarity 0.85)
- One Prompt Fits All: Universal Graph Adaptation for Pretrained Models (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.