Research map / Architectures and efficiency / Attention and transformers

Structure-Guided Masked Autoencoders for Ultra-High Resolution Scientific Image Understanding

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Token merging and masked pretraining · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SGMA, a structure-guided MAE pre-training framework for gigapixel scientific images that pairs a content-adaptive quadtree tokenizer (SGFT) with structure-conditioned masking (Damped Accumulation), reporting large Dice gains over MAE baselines on TEM, XCT and WSI segmentation with up to 24.8x inference speedup.

Problem

Masked autoencoder pre-training and ViT inference are impractical for gigapixel scientific images: random masking ignores multi-scale microstructure, and uniform tokenization yields sequences too long for O(N^2) attention.

Approach

Replace uniform patching with a probabilistic, softmax-sampled content-adaptive quadtree that yields a fixed-length token sequence, and build the MAE masking distribution from a Gaussian-noise canvas accumulated across tree levels (Damped Accumulation), so masks concentrate on structurally informative regions.

Key results

• On SpringXCT-8K, FT-SGMA-SAM reaches 95.68% Dice vs 82.68% for FT-MAE-SAM (+13.00).
• On PAIP-32K, FT-SGMA-SAM2 reaches 83.21% Dice, +16.84 over FT-MAE-SAM2 (66.37%).
• Inference speedup up to 24.8x over MAE-SAM on PAIP-32K (0.3663 s vs 8.9812 s per image), though with a shorter sequence length for SGMA.

Surprising findings

• The paper's own table shows FT-MAE-UNETR (86.12%) outperforming FT-SGMA-UNETR (91.93%) only in the wrong direction; baseline strength varies widely across architectures, and the reported MAE-SAM2 improvement baseline is inconsistent (66.37 vs 85.98 used at different points).
• DA ablation shows near-zero gain for classification (+0.88 top-1 without DA) but +3.00 Dice for segmentation.

Limitations

• Performance depends on Canny edge maps for salience; weak-gradient or texture-defined regions may be poorly tokenized.
• Fixed sequence length N must be tuned per dataset; too small N degrades Dice on heterogeneous inputs (e.g. PAIP-32K with N=1024).
• Speedup comparison uses a deliberately shorter SGMA sequence and different sequence lengths than MAE, so speedup numbers are not strictly matched-compute comparisons.

Experimental setup

Datasets: HydrogelTEM-1K SpringXCT-8K PAIP-32K BTCV KiTS19

Benchmarks: HydrogelTEM-1K SpringXCT-8K PAIP-32K BTCV KiTS19 PAIP-16K classification

Metrics: Dice score Top-1 accuracy inference time per image speedup

Baselines: MAE-SAM MAE-SAM2 MAE-UNETR UNETR U-Net SAM 2 Swin-UNETR HIPT SHF-ViT ViT

Builds on: Masked Autoencoders Segment Anything Model UNETR Adaptive Patching SHF

Closest papers by meaning

• Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology (CVPR 2024 · similarity 0.87)
• Extending Video Masked Autoencoders to 128 frames (NeurIPS 2024 · similarity 0.87)
• ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning (NeurIPS 2025 · similarity 0.87)
• Reconstruction Target Matters in Masked Image Modeling for Cross-Domain Few-Shot Learning (AAAI 2025 · similarity 0.87)
• Self-Guided Masked Autoencoders for Domain-Agnostic Self-Supervised Learning (ICLR 2024 · similarity 0.86)
• R-MAE: Regions Meet Masked Autoencoders (ICLR 2024 · similarity 0.86)