Research map / 3D vision / 3D generation
Learning Multimodal Volumetric Features for Large-Scale Neuron Tracing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Predicts connectivity between over-segmented neuron fragments by fusing a connectivity-aware contrastive volumetric EM embedding (EmbedNet) with point-cloud or voxel morphology, trained on the new FlyTracing dataset from the whole fly brain.
Problem
Reducing human proofreading workload in neuron reconstruction from large EM volumes by automatically predicting whether over-segmented neuron fragments (split errors) belong to the same neuron, especially under imaging artifacts like section missing and misalignment.
Approach
A segment-connectivity-oriented contrastive loss (merge and split terms on mean segment embeddings) trained with a segmentation clustering loss on FFN over-segments learns dense voxel EM embeddings from sparse connectivity annotations; these are fused voxel-wise or per-point with PointNet++ or EdgeNetwork morphology features.
Key results
- PointNet++ with Connect-Embed reaches F1 0.952 on the 3K FlyTracing test blocks, versus 0.891 for PointNet++ alone and 0.911 for EdgeNetwork.
- On misalignment blocks, PointNet++ + Connect-Embed reaches F1 0.884, compared with 0.737 for PointNet++ alone.
- In a FAFB mushroom body tracing experiment, automatic proofreading increases ERL by 28.8 µm (a relative increase of 8.1%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse point-cloud morphology with connectivity-aware embedding instead of using PointNet++ morphology alone | PointNet++ · FlyTracing test blocks (3K) | F1: 0.891 -> 0.952 | Helped |
| Fuse point-cloud morphology with connectivity-aware embedding instead of using EdgeNetwork morphology alone | EdgeNetwork · FlyTracing test blocks (3K) | F1: 0.911 -> 0.925 | Helped |
| Fuse point-cloud morphology with connectivity-aware embedding instead of using PointNet++ morphology alone | PointNet++ · Misalignment | F1: 0.737 -> 0.884 | Helped |
| Fuse point-cloud morphology with connectivity-aware embedding instead of using PointNet++ morphology alone | PointNet++ · Missing sections | F1: 0.781 -> 0.935 | Helped |
| Fuse point-cloud morphology with connectivity-aware embedding instead of using PointNet++ morphology alone | PointNet++ · Mixed degradation | F1: 0.771 -> 0.915 | Helped |
| Use adaptive λ₃ scheduling instead of training with L_seg only (ablation) | EmbedNet · 3 test blocks | Mean rank: 4.07 -> 1.41 | Helped |
| Decrease λ₃ adaptively instead of keeping it fixed at 1 (ablation) | EmbedNet · 3 test blocks | Mean rank: 2.15 -> 1.41 | Helped |
| Decrease λ₃ adaptively instead of keeping it fixed at 0.1 (ablation) | EmbedNet · 3 test blocks | Mean rank: 1.57 -> 1.41 | Helped |
| Remove L_seg from the loss (λ₃=0) instead of using adaptive λ₃ (ablation) | EmbedNet · 3 test blocks | Mean rank: 1.41 -> 2.15 | Hurt |
| Use pairwise segment connectivity prediction for automatic proofreading | PointNet++ + Connect-Embed on FAFB-FFN1 · FAFB Mushroom Body | ERL increase: 28.8μm (a relative increase of 8.1% ) | Helped |
Surprising findings
- Raw voxel intensity combined with point clouds gives only limited improvement, while the segmentation-based embedding from prior work also fails to be robust to imaging artifacts.
- Adaptively decreasing the segmentation clustering loss weight outperforms any fixed weight in the ablation.
Limitations
- Only pairwise connection prediction is studied; multi-candidate reasoning and tracing history are left for future work.
- Training labels come from FFN over-segmentation and skeleton registration, which contain assignment noise that is filtered heuristically.
- Tracing evaluation on FAFB is limited to a single mushroom body subvolume with 166 skeletons.
Experimental setup
Datasets: FlyTracing FAFB CREMI
Benchmarks: FlyTracing test blocks
Metrics: recall precision F1 expected run length (ERL) discriminative rank
Baselines: EdgeNetwork PointNet++ Seg-Embed Intensity Connect-Embed o.
Builds on: PointNet++ EdgeNetwork Flood-filling networks supervised contrastive learning residual symmetric U-Net
Closest papers by meaning
- Cross-Dimension Affinity Distillation for 3D EM Neuron Segmentation (CVPR 2024 · similarity 0.90)
- Efficient Connectivity-Preserving Instance Segmentation with Supervoxel-Based Loss Function (AAAI 2025 · similarity 0.87)
- FISBe: A Real-World Benchmark Dataset for Instance Segmentation of Long-Range Thin Filamentous Structures (CVPR 2024 · similarity 0.85)
- Joint Learning Neuronal Skeleton and Brain Circuit Topology with Permutation Invariant Encoders for Neuron Classification (AAAI 2024 · similarity 0.85)
- Learning to cluster neuronal function (NeurIPS 2025 · similarity 0.83)
- Reproducibility of predictive networks for mouse visual cortex (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.