Research map / Architectures and efficiency / Attention and transformers
Electron Microscopy Images as Set of Fragments for Mitochondrial Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FragViT, a transformer that partitions 3D EM volumes into learned irregular fragments via affinity heads and hierarchically aggregates fragment predictions to voxels, for mitochondria and neuron segmentation.
Problem
Mitochondria segmentation in 3D EM stacks suffers from coarse predictions because CNN and ViT features lie on regular 3D grids that mix foreground and background within cells.
Approach
Interpret the EM volume as a set of learned 3D fragments: each voxel is softly assigned to 27 neighboring token placeholders via a learned affinity, so tokens become homogeneous-semantic fragments; predictions are aggregated back to voxels stage by stage.
Key results
- On MitoEM-R, FragViT reaches 76.9 AP vs 72.8 for ResUNet and 73.1 for Swin UNETR.
- On Lucchi, FragViT achieves 90.5 Jaccard and 95.0 DSC, above ResUNet's 89.5 and 94.5.
- On AC3/AC4, FragViT gets VOI 1.054 and ARAND 0.093, better than PEA (1.205, 0.121).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace regular grid features with learned 3D fragments and hierarchical fragment aggregation | mitochondria segmentation · MitoEM-R validation | AP: 73.1 -> 76.9 | Helped |
| replace regular grid features with learned 3D fragments and hierarchical fragment aggregation | mitochondria segmentation · Lucchi | Jaccard: 89.5 -> 90.5 | Helped |
| replace regular grid features with learned 3D fragments and hierarchical fragment aggregation | neuron instance segmentation · AC3/AC4 | VOI: 1.205 -> 1.054 | Helped |
| add fragment modeling to the transformer baseline (ablation) | FragViT · MitoEM-R validation | AP: 73.0 -> 74.2 | Helped |
| add hierarchical fragment aggregation to fragment modeling (ablation) | FragViT · MitoEM-R validation | AP: 74.2 -> 76.9 | Helped |
| use a 3 × 3 × 3 neighborhood instead of a 1 × 3 × 3 neighborhood (ablation) | FragViT · MitoEM-R validation | AP: 75.3 -> 76.9 | Helped |
| use a 3 × 3 × 3 neighborhood instead of a 6-nearest neighborhood (ablation) | FragViT · MitoEM-R validation | AP: 76.2 -> 76.9 | Helped |
| use full hierarchical aggregation from stage 4 to stage 1 rather than direct stage 4 to stage 1 aggregation (ablation) | FragViT · MitoEM-R validation | AP: 74.2 -> 76.9 | Helped |
| use full hierarchical aggregation from stage 4 to stage 1 rather than stage 3 to stage 2 to stage 1 (ablation) | FragViT · MitoEM-R validation | AP: 74.7 -> 76.9 | Helped |
| use full hierarchical aggregation from stage 4 to stage 1 rather than stage 4 to stage 3 to stage 2 (ablation) | FragViT · MitoEM-R validation | AP: 75.4 -> 76.9 | Helped |
Surprising findings
- One-step aggregation from stage 4 directly to voxels performed worse (74.2 AP) than the full stage-by-stage hierarchical path (76.9 AP).
Limitations
- Evaluated only on fixed, small benchmark splits; no broader generalization study.
- Instance segmentation relies on a heuristic seeded watershed post-processing step.
- Compares against reproduced baselines (marked with †) whose exact settings are not fully detailed.
Experimental setup
Datasets: MitoEM Lucchi AC3 AC4
Benchmarks: MitoEM-R MitoEM-H Lucchi AC3/AC4
Metrics: AP AP50 AP75 APs APm APl Jaccard DSC VOI ARAND
Baselines: U2D-B U3D-A U3D-BC Nightingale CLMS ResUNet MaskFormer Mask2Former UNETR Swin UNETR SuperHuman MALA
Builds on: Vision Transformer Swin UNETR affinity learning (PEA) Mask2Former Dice loss
Closest papers by meaning
- Alleviate and Mining: Rethinking Unsupervised Domain Adaptation for Mitochondria Segmentation from Pseudo-Label Perspective (AAAI 2025 · similarity 0.87)
- Evidential Uncertainty-Guided Mitochondria Segmentation for 3D EM Images (AAAI 2024 · similarity 0.87)
- $\epsilon$-Seg: Sparsely Supervised Semantic Segmentation of Microscopy Data (NeurIPS 2025 · similarity 0.86)
- Cross-Dimension Affinity Distillation for 3D EM Neuron Segmentation (CVPR 2024 · similarity 0.85)
- Rethinking Decoder Design: Improving Biomarker Segmentation Using Depth-to-Space Restoration and Residual Linear Attention (CVPR 2025 · similarity 0.82)
- Micro-Bench: A Microscopy Benchmark for Vision-Language Understanding (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.