Research map / Time series / Foundation models for time series
Brain Harmony: A Multimodal Foundation Model Unifying Morphology and Function into 1D Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
placeholder
Problem
Existing brain foundation models handle either structural MRI or fMRI dynamics, ignore how brain morphology constrains function, and cannot accept fMRI with heterogeneous repetition times (TRs).
Approach
Pre-align fMRI positional embeddings with geometric harmonics of the cortical surface, use a TR-adaptive patch embedding (TAPE) with pseudoinverse weight resizing and temporal downsampling augmentation, and fuse T1 and fMRI latents into a small set of learnable 1D brain hub tokens.
Key results
- BrainHarmonix reaches 72.63% F1 on ABIDE-I and 70.09% accuracy on ADHD-200, outperforming prior baselines on neurodevelopmental tasks.
- On PPMI, BrainHarmonix achieves 64.34% accuracy and on ADNI 64.65% accuracy, the best among compared models.
- On HCP-A cognition prediction, BrainHarmonix achieves correlation 0.42 versus 0.28 for BrainMass, though its MAE (6.56) is worse than Brain-JEPA (5.62) and BrainMVP-2 (5.39).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add geometric pre-alignment to fMRI representations (ablation) | BrainHarmonix-F · ADNI | accuracy: 61.35 -> 61.62 | Helped |
| add multi-TR downsampling data augmentation (ablation) | BrainHarmonix · ADNI | accuracy: 62.94 -> 64.65 | Helped |
| add multi-TR downsampling data augmentation (ablation) | BrainHarmonix-F · ADNI | accuracy: 60.11 -> 61.62 | Helped |
| fuse structural and functional representations rather than use function-only BrainHarmonix-F (ablation) | BrainHarmonix · ADNI | accuracy: 61.62 -> 64.65 | Helped |
| pretrain BrainHarmonix-F with ABCD in addition to UKB (ablation) | BrainHarmonix-F · ADNI | accuracy: 60.67 -> 61.62 | Helped |
| pretrain BrainHarmonix-F with ABCD in addition to UKB (ablation) | BrainHarmonix-F · ADNI | F1: 63.34 -> 64.80 | Helped |
| increase the harmonizer size from 22M to 307M parameters (ablation) | BrainHarmonix · ABIDE-II | accuracy: 64.06 -> 66.95 | Helped |
| increase the harmonizer size from 22M to 307M parameters (ablation) | BrainHarmonix · ADHD-200 | accuracy: 69.39 -> 70.40 | Helped |
| increase the number of 1D tokens from 128 to 256 (ablation) | BrainHarmonix · ABIDE-II | accuracy: 66.67 -> 66.96 | Helped |
| increase the number of 1D tokens from 128 to 256 (ablation) | BrainHarmonix · ADHD-200 | accuracy: 70.09 -> 70.41 | Helped |
| replace the function-only representation with multimodal structural-functional fusion (ablation) | BrainHarmonix · ADHD-200 | F1: 68.75 -> 66.72 | Hurt |
Surprising findings
- BrainHarmonix's MAE on HCP-A (6.56) is worse than several baselines even though its correlation is best.
- Five of 128 hub tokens were cross-modal, while 93 attended only to fMRI, suggesting fMRI dominates the fused representation.
Limitations
- Age coverage is limited (UKB 44-83, ABCD 8-11), lacking infancy and young adulthood.
- Unimodal encoders are frozen during fusion, and joint optimization was not explored.
- Performance is weaker in low-sample / few-shot settings and degraded by T1 motion artifacts in ADHD.
Experimental setup
Datasets: UK Biobank ABCD ABIDE-I ABIDE-II ADHD-200 PPMI ADNI HCP-A Schaefer-400
Benchmarks: ABIDE-I ABIDE-II ADHD-200 PPMI ADNI HCP-A
Metrics: ACC F1 MAE Pearson correlation
Baselines: BrainMVP BrainNetCNN BrainGNN BrainNetTF BrainMass BrainLM Brain-JEPA
Builds on: Brain-JEPA masked autoencoder BrainLM BrainMVP pseudoinverse resize
Closest papers by meaning
- Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG (NeurIPS 2026 · similarity 0.88)
- BrainWorld: A Structural-Prior-Conditioned Generative Model for Whole-Brain 4D fMRI Dynamics (NeurIPS 2026 · similarity 0.87)
- BrainLM: A foundation model for brain activity recordings (ICLR 2024 · similarity 0.87)
- MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding (NeurIPS 2026 · similarity 0.87)
- NeuralFlix: A Simple While Effective Framework for Semantic Decoding of Videos from Non-invasive Brain Recordings (AAAI 2025 · similarity 0.87)
- BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.