Research map / Graph learning / GNN architectures
Mixture of Weak and Strong Experts on Graphs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mowst, a mixture of a weak MLP expert and a strong off-the-shelf GNN expert, where a confidence gate based on the MLP's prediction dispersion decides per node whether to use the MLP or fall back to the GNN.
Problem
GNNs apply global, uniform aggregation to all nodes even though nodes differ in whether self-features or neighborhood structure drive their labels; heterophilous neighbors can add noise, and MoE over GNNs is costly and hard to optimize.
Approach
Mix an MLP (weak expert) and a GNN (strong expert) at the output level, with a confidence C = G(D(p)) computed from the dispersion (variance or negative entropy) of only the MLP's logits; the MLP is used with probability C, else the GNN is run. Training alternates expert updates or jointly minimizes the expected loss, and the design is biased toward the GNN.
Key results
- Mowst on GCN improves over the GCN baseline on all six benchmarks, e.g., +0.76 on Flickr (54.62 vs 53.86) and +1.02 on Penn94 (83.19 vs 82.17).
- Mowst on GIN gains up to +22.74 on Pokec (76.11 vs 53.37), where the GIN baseline converges poorly.
- Mowst on H2GCN improves accuracy on Penn94, Pokec and Twitch by +0.68, +2.13 and +0.33 respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add an MLP expert with confidence-based gating to GCN | GCN · Flickr, Products, arXiv, Penn, Pokec, Twitch | accuracy: +0.76; +0.85; +0.64; +1.02; +0.29; +0.83 | Helped |
| Add an MLP expert with confidence-based gating to GIN | GIN · Flickr, arXiv, Penn, Pokec, Twitch | accuracy: +1.77; +2.04; +1.88; +22.74; +2.56 | Helped |
| Add an MLP expert with confidence-based gating to GIN-skip | GIN-skip · Flickr, arXiv, Penn, Pokec, Twitch | accuracy: +0.49; +0.51; +0.88; +3.41; +0.64 | Helped |
| Add an MLP expert with confidence-based gating to GraphSAGE | GraphSAGE · Flickr, Products, arXiv, Penn, Pokec, Twitch | accuracy: +0.39; +0.88; +0.55; +2.03; +1.33; +1.05 | Helped |
| Add an MLP expert with confidence-based gating to H2GCN | H2GCN · Penn, Pokec, Twitch | accuracy: +0.68; +2.13; +0.33 | Helped |
| Train both experts simultaneously instead of using in-turn training (ablation) | Mowst-GCN · Flickr, Pokec, Twitch | accuracy: 54.62 -> 53.47; 77.12 -> 76.62; 63.74 -> 63.44 | Hurt |
| Use the Mowst+ prediction-mixture loss instead of the Mowst loss (ablation) | Mowst-GCN · Flickr, Pokec, Twitch | accuracy: 54.62 vs. 53.94; 77.12 vs. 77.28; 63.74 vs. 63.59 | Mixed |
| Replace the weak MLP expert with a shallow GCN expert (ablation) | Mowst-GCN · Flickr, Products, arXiv, Penn, Pokec, Twitch | accuracy: 53.86 -> 54.19; 75.64 -> 74.47; 71.74 -> 71.92; 82.17 -> 82.38; 76.01 -> 76.51; 62.42 -> 63.36 | Mixed |
| Add skip connections to GCN layers (ablation) | GCN · Flickr, arXiv, Penn, Pokec, Twitch | accuracy: 53.86 -> 52.98; 71.74 -> 69.56; 82.17 -> 76.58; 76.01 -> 73.46; 62.42 -> 61.05 | Hurt |
| Add skip connections to GIN layers (ablation) | GIN · arXiv, Pokec, Twitch | accuracy: 69.39 -> 71.28; 53.37 -> 76.29; 61.76 -> 64.27 | Mixed |
| Add an MLP expert to GIN-skip (ablation) | GIN-skip · arXiv, Pokec, Twitch | accuracy: 71.28 -> 71.79; 76.29 -> 79.70; 64.27 -> 64.91 | Mixed |
Surprising findings
- Adding a shallow GCN as the weak expert (Weak-Strong GCN) gives smaller gains than an MLP weak expert and even degrades Flickr accuracy.
- A GCN-skip variant consistently degrades GCN performance, whereas the MLP-expert Mowst on GCN improves it.
Limitations
- Fair comparison is ambiguous: the paper prioritizes computation cost over model size, so Mowst has more parameters than its GNN baseline.
- Gains depend on dataset and backbone; the multi-expert generalization is not comprehensively evaluated.
- Experiments are full-batch with no neighborhood sampling, which limits scalability claims on very large graphs.
Experimental setup
Datasets: Flickr ogbn-arxiv ogbn-products Penn94 Pokec Twitch
Benchmarks: Flickr ogbn-arxiv ogbn-products Penn94 Pokec Twitch
Metrics: accuracy
Baselines: MLP GCN GraphSAGE GAT GIN GPR-GNN H2GCN AdaGCN GraphMoE GCN-skip GIN-skip
Builds on: Mixture of Experts GraphMoE GPR-GNN H2GCN GLNN
Closest papers by meaning
- Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling (NeurIPS 2023 · similarity 0.90)
- Deep Ensembles for Graphs with Higher-order Dependencies (ICLR 2023 · similarity 0.88)
- MORGAN: To Bridge Mixture of Experts and Spectral Graph Neural Network (AAAI 2026 · similarity 0.88)
- Graph Sparsification via Mixture of Graphs (ICLR 2025 · similarity 0.87)
- Graph Neural Networks Are More Than Filters: Revisiting and Benchmarking from A Spectral Perspective (ICLR 2025 · similarity 0.86)
- Two Heads Are Better Than One: Boosting Graph Sparse Training via Semantic and Topological Awareness (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.