Research map / Architectures and efficiency / Attention and transformers
Boltzmann Attention Sampling for Image Analysis with Small Objects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BoltzFormer, a transformer decoder that samples sparse image regions for cross-attention via a Boltzmann distribution with an annealing temperature, evaluated for text-prompted small-object segmentation on medical images.
Problem
Transformer attention wastes computation on irrelevant pixels when segmenting or detecting small objects (often under 0.1% of the image) with only a text prompt, and rigid sparse-attention schemes fit poorly.
Approach
Each query defines a Boltzmann distribution over image positions, with temperature decaying across layers (tau_l = tau_0/(1+l)); queries sample a sparse patch set and cross-attend only to it, with a PiGMA module aggregating query predictions.
Key results
- BoltzFormer+FocalL reaches 75.2 mean Dice across seven benchmarks, versus 73.8 for SEEM+Hiera-BP and 73.0 for BiomedParse.
- On small objects (<1% image area), BoltzFormer scores 71.4 Dice versus 68.9 for SEEM.
- Boltzmann sampling scores 73.7 average Dice versus 72.4 for a fixed 0.5 threshold mask and 71.2 for full attention.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace full attention with Boltzmann attention sampling (ablation) | BoltzFormer · benchmark suite | average Dice score (%): 71.2 (0.43) -> 73.7 (0.38) | Helped |
| replace fixed 0.5 threshold masking with Boltzmann attention sampling (ablation) | BoltzFormer · benchmark suite | average Dice score (%): 72.4 (0.41) -> 73.7 (0.38) | Helped |
| vary base sampling temperature τ0; τ0 = 1 performs best among tested settings (ablation) | BoltzFormer · benchmark suite | average Dice score (%): 73.6 (0.40); 73.0 (0.43); 73.7 (0.38); 72.2 (0.42) | Mixed |
| vary the number of sampled visual features per query per layer; 10% performs best among tested settings (ablation) | BoltzFormer · benchmark suite | average Dice score (%): 72.9 (0.40); 73.7 (0.38); 72.4 (0.42); 73.6 (0.40) | Mixed |
| use an ensemble of queries instead of a single query (ablation) | BoltzFormer · benchmark suite | average Dice score (%): 72.8 (0.42) -> 73.8 (0.38) | Helped |
| increase the query ensemble from 10 to 32 queries (ablation) | BoltzFormer · benchmark suite | average Dice score (%): 73.8 (0.38) -> 73.8 (0.38) | No clear effect |
| condition the query prior on the input text (ablation) | BoltzFormer · benchmark suite | average Dice score (%): 72.3 (0.44) -> 73.7 (0.38) | Helped |
| add pixel-grounded correction to query ensemble prediction (ablation) | BoltzFormer PiGMA · benchmark suite | average Dice score (%): 73.2 (0.43) -> 73.7 (0.38) | Helped |
| use BoltzFormer instead of the strongest baseline on small objects | segmentation models · small objects (< 1% image area) | Dice score: 68.9 (0.50) -> 71.4 (0.44) | Helped |
Surprising findings
- Full attention with a fixed threshold mask is outperformed by probabilistic sampling, and SAM 2 decoder underperforms SAM decoder with Hiera-BP backbone on average.
Limitations
- Failure cases include extremely small targets, low contrast, and presence of other plausible objects (about 1.4% complete misses).
- Evaluated only on medical segmentation benchmarks with text prompts from BiomedParse GPT-4 enriched data.
Experimental setup
Datasets: Medical Segmentation Decathlon LIDC-IDRI AMOS22 BiomedParse GPT-4 enriched text prompts
Benchmarks: LIDC AMOS-CT AMOS-MRI MSD-Lung MSD-Pancreas MSD-HepaticVessel MSD-Colon
Metrics: Dice score
Baselines: nnU-Net BiomedParse SAM SAM 2 SEEM Mask2Former
Builds on: Mask2Former SEEM SAM BiomedParse
Closest papers by meaning
- CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up (NeurIPS 2025 · similarity 0.84)
- TGBFormer: Transformer-GraphFormer Blender Network for Video Object Detection (AAAI 2025 · similarity 0.84)
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.84)
- SparseFormer: Sparse Visual Recognition via Limited Latent Tokens (ICLR 2024 · similarity 0.84)
- Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · similarity 0.84)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.