Research map / Image generation / Fast diffusion sampling
TractoTransformer: Diffusion MRI Streamline Tractography using CNN and Transformer Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TractoTransformer, a CNN-plus-decoder-only-Transformer model that predicts conditional fODF directions auto-regressively along streamlines for diffusion MRI tractography.
Problem
White matter tractography from noisy, ambiguous dMRI measurements, where crossing, merging and fanning fiber configurations degrade per-voxel orientation predictions that ignore spatial or sequential structure.
Approach
Treats tractography as autoregressive sequence modeling: a 3D CNN embeds local dMRI neighborhoods of each streamline point, and a decoder-only Transformer predicts a smoothed distribution over 724 sphere directions plus an end-of-fiber class, conditioned on the trajectory so far.
Key results
- On the ISMRM Tractometer benchmark, TractoTransformer reports VC 82%, OL 84%, and F1 75%, the highest among compared methods, with OR 31%.
- Ablation: removing the 3D CNN drops VC from 81.51% to 69.86% (-11.65 points).
- Full-brain inference on one subject takes about 2.5 hours on a single V100 GPU, or about 43 minutes across 4 GPUs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TractoTransformer instead of DeepTract | ISMRM 2015 Tractography Challenge · ISMRM 2015 Tractography Challenge | VC (%): 71 -> 82 | Helped |
| Use TractoTransformer instead of RF | ISMRM 2015 Tractography Challenge · ISMRM 2015 Tractography Challenge | OL (%): 75 -> 84 | Helped |
| Use TractoTransformer instead of CTtrack | ISMRM 2015 Tractography Challenge · ISMRM 2015 Tractography Challenge | OR (%): 16 -> 31 | Hurt |
| Use TractoTransformer instead of DeepTract | ISMRM 2015 Tractography Challenge · ISMRM 2015 Tractography Challenge | F1 (%): 70 -> 75 | Helped |
| Add the CNN3D module (ablation) | TractoTransformer · ISMRM 2015 Tractography Challenge | VC (%): 69.86 -> 81.51 | Helped |
| Add reverse-streamline augmentation (ablation) | TractoTransformer · ISMRM 2015 Tractography Challenge | F1 (%): 73.76 -> 74.78 | Helped |
| Add label smoothing (ablation) | TractoTransformer · ISMRM 2015 Tractography Challenge | F1 (%): 74.85 -> 74.78 | No clear effect |
Surprising findings
- Removing label smoothing leaves F1 essentially unchanged (74.85 vs 74.78), and the paper's OR for CTtrack (16%) is better than TractoTransformer's (31%).
Limitations
- CNN3D raises training memory cost, limiting how many subjects fit in memory per batch (ten TractoInferno subjects).
- Full-brain inference is slow (about 2.5 hours per subject on one V100 GPU).
- In vivo evaluation is qualitative only, on one test subject, trained on just ten subjects.
Experimental setup
Datasets: ISMRM 2015 Tractography Challenge phantom TractoInferno
Benchmarks: ISMRM 2015 Tractometer TractoInferno (qualitative)
Metrics: VC OL OR F1
Baselines: RF LearnToTrack DeepTract Entrack TrackToLearn CTtrack
Builds on: Transformer decoder CTtrack DeepTract LearnToTrack
Closest papers by meaning
- Equivariant spatio-hemispherical networks for diffusion MRI deconvolution (NeurIPS 2024 · similarity 0.88)
- Self-Supervised Diffusion MRI Denoising via Iterative and Stable Refinement (ICLR 2025 · similarity 0.84)
- MAGNET: Manifold-Aware Graph Diffusion Network for Connectome Generation (NeurIPS 2026 · similarity 0.83)
- Patch2Self2: Self-supervised Denoising on Coresets via Matrix Sketching (CVPR 2024 · similarity 0.82)
- $\textit{NeuroPath}$: A Neural Pathway Transformer for Joining the Dots of Human Connectomes (NeurIPS 2024 · similarity 0.82)
- A Steerable Deep Network for Model-Free Diffusion MRI Registration (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.