Research map / Label-efficient and robust learning / Knowledge distillation
Pre-Training Graph Contrastive Masked Autoencoders are Strong Distillers for EEG
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EEG-DisGCMAE, a unified graph contrastive masked autoencoder pre-training and topology distillation framework that transfers knowledge from high-density to low-density EEG graphs for clinical classification.
Problem
Leveraging abundant unlabeled high-density EEG and high-density labeled data to improve classification on scarce, low-density EEG data, where electrodes are missing.
Approach
Unifies graph contrastive pre-training and graph masked autoencoder pre-training so each supervises the other (reconstructing contrastive samples and contrasting reconstructions), jointly pre-trains HD teacher and LD student encoders, and adds a graph topology distillation loss that uses removed electrodes to define positive and negative node pairs.</, "core_idea_end">
Key results
- Ours-Large (Gformer) on LD EEG reaches 87.4/87.8 AUROC/ACC on HBN MDD HD and 84.8/86.9 on LD, outperforming GraphMAE2 (83.5/85.7 HD, 81.3/83.0 LD).
- Ours-Tiny (DGCNN) with 1.3M parameters performs comparably to larger pre-training baselines with 5.7M parameters, per the paper's efficiency analysis.
- Under Gaussian noise and electrode dropping perturbations on HBN MDD, the proposed model keeps 83.7% and 84.0% versus GraphMAE's 78.5% and 77.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GCMAE-PT instead of GCL-PT, GMAE-PT, or their sequential combination (ablation) | GFormer-Tiny student · HBN MDD LD | AUROC/ACC: 75.6/78.1 -> 77.6/79.4 | Helped |
| Combine logits distillation and GTD loss (ablation) | GFormer-Tiny student · HBN MDD LD | AUROC/ACC: 72.7/75.4 -> 75.0/79.2 | Helped |
| Add logits distillation (ablation) | GFormer-Tiny student · HBN MDD LD | AUROC/ACC: 72.7/75.4 -> 73.6/77.3 | Helped |
| Add GTD loss (ablation) | GFormer-Tiny student · HBN MDD LD | AUROC/ACC: 72.7/75.4 -> 73.8/78.5 | Helped |
| Use GCL-PT instead of GCMAE-PT (ablation) | GFormer-Tiny student · HBN MDD LD | AUROC/ACC: 77.6/79.4 -> 74.3/75.9 | Hurt |
| Use GMAE-PT instead of GCMAE-PT (ablation) | GFormer-Tiny student · HBN MDD LD | AUROC/ACC: 77.6/79.4 -> 75.1/76.6 | Hurt |
| Sequentially combine GCL-PT and GMAE-PT instead of using GCMAE-PT (ablation) | GFormer-Tiny student · HBN MDD LD | AUROC/ACC: 77.6/79.4 -> 75.6/78.1 | Hurt |
| Reduce electrode density from HD to LD (ablation) | EEG classification models · HBN MDD | AUROC/ACC: 80.4/83.6 -> 72.6/76.4 | Hurt |
| Use a large rather than tiny GFormer (ablation) | EEG-DisGCMAE · HBN MDD HD | AUROC/ACC: 85.3/86.8 -> 87.4/87.8 | Helped |
| Use the linear kernel instead of other tested similarity kernels for GTD (ablation) | GTD distillation · EEG distillation | distillation effect | Helped |
| Use GTD for H2H rather than H2L distillation (ablation) | GTD distillation · H2H versus H2L | optimization convergence | Hurt |
| Use GCMAE-PT reconstruction rather than vanilla GMAE-PT reconstruction at 50% masking (ablation) | EEG graph reconstruction · EMBARC MDD severity | MSE loss: 0.31 -> 0.17 | Helped |
Surprising findings
- The GTD loss is less effective in the high-to-high special case where no electrodes are removed, with a less pronounced optimization decrease.
- Sequentially combining contrastive and masked autoencoder pre-training does not outperform the unified GCMAE-PT framework, which the authors interpret as a sign that mutual supervision matters.
Limitations
- Downstream labeled datasets are small (roughly 240-490 subjects per task), and results rely on 10-fold cross-validation.
- Node selection for the GTD loss depends on the teacher's learned adjacency, and the loss is less effective in the high-to-high case.
Experimental setup
Datasets: EMBARC HBN SEED
Benchmarks: HBN MDD HBN ASD EMBARC Sex EMBARC Severity SEED
Metrics: AUROC accuracy
Baselines: MLP LSTM GCN GFormer Hyper-GCN EEGNet EEG Conformer RGNN DGCNN GCC GRACE GraphCL
Builds on: GraphMAE MoCo GCC GraphCL CMAE
Closest papers by meaning
- Standing on the Shoulders of Giants: Rethinking EEG Foundation Model Pretraining via Multi-Teacher Distillation (NeurIPS 2026 · similarity 0.88)
- EEGPT: Pretrained Transformer for Universal and Reliable Representation of EEG Signals (NeurIPS 2024 · similarity 0.85)
- CiTrus: Squeezing Extra Performance out of Low-data Bio-signal Transfer Learning (AAAI 2025 · similarity 0.85)
- MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding (NeurIPS 2026 · similarity 0.84)
- What do EEG Foundation Models Capture from Human Brain Signals? (NeurIPS 2026 · similarity 0.84)
- B[FM]$^2$: Brain Foundation Model via Flow Matching with SplitUNet (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.