Research map / Detection and segmentation / Open-set and few-shot detection
Bringing SAM to new heights: leveraging elevation data for tree crown segmentation from drone imagery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks SAM-based and Mask R-CNN-based methods for tree crown instance segmentation in drone imagery across plantation, temperate and tropical forests, and proposes BalSAM, which adds DSM elevation information to RSPrompter-style prompt learning.
Problem
Automatic tree crown instance segmentation (with species classification) from high-resolution drone RGB imagery is understudied due to scarce individual-tree labels; it is unclear whether SAM can be used out-of-the-box or adapted, and whether free photogrammetry DSM data help.
Approach
BalSAM extends RSPrompter by fusing a trainable DSM encoder's embedding with the image embedding (element-wise sum) and feeding it as dense prompt to SAM's mask decoder, with the image encoder and mask decoder kept frozen and the prompter tuned.
Key results
- On Quebec Plantations, BalSAM reaches 54.40 mAP and 64.84 wmAP (multi-class), the best wmAP, versus 62.37 for RSPrompter and 60.32 for Mask R-CNN+DSM.
- On SBL, RSPrompter (33.59 single-class mAP) and BalSAM (33.55) outperform Mask R-CNN (26.16 without ImageNet pre-training / 32.44 with), and BalSAM has the best single-class mIoU (66.02).
- Out-of-the-box SAM with prompts achieves only 8-12 single-class mAP across datasets, below custom Mask R-CNN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DSM as a fourth input channel to ImageNet-pretrained Mask R-CNN (ablation) | Mask R-CNN · Quebec Plantations test | wmAP: 58.30 -> 60.32 | Helped |
| Add DSM as a fourth input channel to ImageNet-pretrained Mask R-CNN (ablation) | Mask R-CNN · SBL test | wmAP: 27.27 -> 26.82 | Hurt |
| Add DSM as a fourth input channel to ImageNet-pretrained Mask R-CNN (ablation) | Mask R-CNN · BCI test | wmAP: 10.33 -> 10.50 | Helped |
| Add ImageNet pretraining to the Mask R-CNN backbone (ablation) | Mask R-CNN · Quebec Plantations test | wmAP: 55.75 -> 58.30 | Helped |
| Add ImageNet pretraining to the Mask R-CNN backbone (ablation) | Mask R-CNN · SBL test | wmAP: 22.45 -> 27.27 | Helped |
| Add DSM input to Mask R-CNN (ablation) | Mask R-CNN · SBL test | multi-class mAP: 21.38 -> 20.87 | No clear effect |
| Replace RGB input with DSM-only input (ablation) | Mask R-CNN · SBL and Quebec Plantations | mIoU | Mixed |
| Use DSM vertical and horizontal gradients as two extra input channels (ablation) | Mask R-CNN · SBL test | performance | No clear effect |
| Add an extra fully connected layer to the box predictor and classification head (ablation) | Mask R-CNN · SBL test | performance | No clear effect |
| Encode DSM with a CNN before providing it to Mask R-CNN (ablation) | Mask R-CNN · BCI test | multi-class mAP: 6.34 -> 8.30 | Helped |
| Use weighted cross-entropy instead of standard cross-entropy (ablation) | SBL classification models · SBL test | performance | Hurt |
| Use focal loss instead of standard cross-entropy (ablation) | SBL classification models · SBL test | performance | Hurt |
Surprising findings
- Prompting SAM with boxes or masks from a trained Mask R-CNN degrades performance relative to the Mask R-CNN predictions themselves.
- Mask2Former does not outperform Mask R-CNN on the Quebec Plantations dataset despite higher capacity.
Limitations
- RSPrompter and BalSAM show higher variance across seeds.
- Long-tailed class imbalance is not fully addressed; classification in dense forests and tropical forests remains hard.
- DSM at 1 m vertical resolution on BCI and DSM is less informative in closed canopies.
Experimental setup
Datasets: Quebec Plantations Quebec Trees (SBL) BCI crown maps
Benchmarks: Quebec Plantations test SBL test BCI test
Metrics: mAP wmAP mIoU
Baselines: SAM automatic SAM+DSM prompts Mask R-CNN Faster R-CNN+SAM Mask R-CNN+SAM Mask2Former RSPrompter
Builds on: Segment Anything Model RSPrompter Mask R-CNN Mask2Former
Closest papers by meaning
- OAM-TCD: A globally diverse dataset of high-resolution tree cover maps (NeurIPS 2024 · similarity 0.88)
- FoMo: Multi-Modal, Multi-Scale and Multi-Task Remote Sensing Foundation Models for Forest Monitoring (AAAI 2025 · similarity 0.85)
- ROS-SAM: High-Quality Interactive Segmentation for Remote Sensing Moving Object (CVPR 2025 · similarity 0.84)
- ForestLPR: LiDAR Place Recognition in Forests Attentioning Multiple BEV Density Images (CVPR 2025 · similarity 0.84)
- ZoRI: Towards Discriminative Zero-Shot Remote Sensing Instance Segmentation (AAAI 2025 · similarity 0.84)
- Deep Change Monitoring: A Hyperbolic Representative Learning Framework and a Dataset for Long-term Fine-grained Tree Change Detection (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.