Research map / Detection and segmentation / Open-set and few-shot detection
GraCo: Granularity-Controllable Interactive Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GraCo, a granularity-controllable interactive segmentation model that takes a granularity parameter as input, trained with automatically generated mask-granularity pairs from a SimpleClick-based any-granularity mask generator and LoRA-based learning.
Problem
Interactive segmentation from sparse clicks is spatially ambiguous; single-granularity models ignore ambiguity while multi-granularity models (e.g., SAM) give redundant outputs with limited control, and annotating multi-granularity masks is costly.
Approach
Add a discrete granularity embedding as an extra prompt to a pre-trained IS model, and generate mask-granularity training pairs automatically by simulating click loops with the pre-trained model and estimating granularity from scale and semantic (probability peak difference) cues, then fine-tune with LoRA.
Key results
- On part-level PascalPart NoC@85, GraCo w/ GT+AGG (ViT-L) reaches 3.87 vs SimpleClick ViT-L 10.23 (single-granularity) and SAM ViT-L 13.15.
- GraCo w/ AGG (ViT-L) achieves NoC@85 of 1.18 on GrabCut and 1.23 on Berkeley, outperforming SAM ViT-L (1.86 and 1.84).
- On a 4987-mask SA-1B subset, GraCo w/ AGG (ViT-L) gets NoC@90 of 3.96 vs SAM ViT-L 3.31 and SimpleClick ViT-L 6.74, showing comparable performance to SAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add granularity-controllable learning trained with GT and AGG mask-granularity pairs | SimpleClick ViT-L · GrabCut | NoC@85: 1.38 -> 1.18 | Helped |
| Add granularity-controllable learning trained with GT and AGG mask-granularity pairs | SimpleClick ViT-L · Berkeley | NoC@85: 1.40 -> 1.17 | Helped |
| Add granularity-controllable learning trained with GT and AGG mask-granularity pairs | SimpleClick ViT-L · SBD | NoC@85: 2.69 -> 2.69 | No clear effect |
| Add granularity-controllable learning trained with GT and AGG mask-granularity pairs | SimpleClick · SBD | NoC@90: 4.46 -> 3.96 | Helped |
| Add granularity-controllable learning trained with GT and AGG mask-granularity pairs | SimpleClick ViT-L · DAVIS | NoC@85: 4.12 -> 3.87 | Helped |
| Add granularity-controllable learning trained with GT and AGG mask-granularity pairs | SimpleClick ViT-L · DAVIS | NoC@90: 5.39 -> 4.83 | Helped |
| Add granularity-controllable learning trained with part ground-truth mask-granularity pairs | SimpleClick ViT-L · PascalPart | NoC@85: 7.28 -> 5.81 | Helped |
| Add granularity-controllable learning trained with GT and AGG mask-granularity pairs | SimpleClick ViT-L · PartImageNet | NoC@85: 5.81 -> 4.92 | Helped |
| Add granularity embedding (ablation) | GCL with LoRA, SimpleClick ViT-B · GrabCut | NoC@85: 3.24 -> 1.46 | Helped |
| Add LoRA instead of full-parameter fine-tuning (ablation) | GCL with granularity embedding, SimpleClick ViT-B · GrabCut | NoC@85: 2.14 -> 1.46 | Hurt |
| Add granularity embedding (ablation) | GCL without LoRA, SimpleClick ViT-B · GrabCut | NoC@85: 3.56 -> 2.14 | Helped |
Surprising findings
- SAM underperforms single-granularity SimpleClick on several object benchmarks in NoC despite its multi-granularity output, attributed to a bias toward small components.
- Fine-tuning SimpleClick on part annotations weakens object-level performance while giving only marginal part-level gains.
Limitations
- Granularity is a user-specified scalar discretized to 0.1 steps and the optimal value for parts varies by part category, so users must estimate it.
- Automatically generated proposals are not verified against true part annotations, and granularity estimation relies on heuristic weighting (lambda=0.5).
- Gains over SAM on SA-1B are only comparable (not superior) at NoC@90.
Experimental setup
Datasets: SBD PascalPart PartImageNet GrabCut Berkeley DAVIS SA-1B
Benchmarks: GrabCut Berkeley SBD DAVIS PascalPart PartImageNet SA-1B (first 1000 images subset)
Metrics: NoC@85 NoC@90 IoU@1
Baselines: SimpleClick SAM FocalClick RITM CDNet GPCIS FocusCut BRS
Builds on: SimpleClick SAM RITM LoRA
Closest papers by meaning
- Universal Segmentation at Arbitrary Granularity with Language Instruction (CVPR 2024 · similarity 0.85)
- Multi-Granularity Video Object Segmentation (AAAI 2025 · similarity 0.85)
- AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation (ICLR 2024 · similarity 0.84)
- Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.84)
- Instruction-guided Multi-Granularity Segmentation and Captioning with Large Multimodal Model (AAAI 2025 · similarity 0.84)
- Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.