Research map / Detection and segmentation / Open-set and few-shot detection
SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SaFiRe, a Mamba-based referring image segmentation model that alternates Saccade (global norm-adaptation) and Fixation (window-wise scanning with repeated text tokens) layers, and introduces the test-only aRefCOCO benchmark for ambiguous expressions.
Problem
Referring image segmentation methods oversimplify the task to keyword/concept matching and fail under referential ambiguity, such as object-distracting expressions with multiple noun phrases and category-implicit expressions lacking explicit object class.
Approach
Mimics a two-phase human cognitive process via alternating Saccade (DiT-style global norm adaptation of image features with pooled text) and Fixation (window-grouped VMamba scanning where text tokens are re-inserted after each image window), exploiting Mamba's order-sensitive scan-then-update property.
Key results
- Single-dataset training: SaFiRe reaches 67.07 oIoU on RefCOCOg val(U), outperforming MagNet (65.36) and other listed methods.
- Mixed-data training: SaFiRe reaches 75.8 oIoU on RefCOCOg val(U) and 81.4 on RefCOCO val, exceeding u-LLaVA (74.8) and GSVA among others.
- Inference: 8.26 FPS at 384 GFLOPs, claimed highest throughput and lowest compute among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SaFiRe instead of the strongest listed single-dataset baseline | RIS with Swin-B and BERT · RefCOCOg val(U) | oIoU: 64.21 -> 67.07 | Helped |
| Use SaFiRe instead of the strongest listed single-dataset baseline | RIS with Swin-B and BERT · RefCOCOg test(U) | oIoU: 66.22 -> 66.87 | Helped |
| Use SaFiRe instead of the strongest listed single-dataset baseline | RIS with Swin-B and BERT · RefCOCO+ val, testA, and testB | oIoU: 66.16 -> 66.39; 71.84 -> 71.96; 58.14 -> 58.47 | Helped |
| Use SaFiRe instead of the strongest listed single-dataset baseline | RIS with Swin-B and BERT · RefCOCO val, testA, and testB | oIoU: 75.24 -> 75.34; 78.24 -> 78.52; 71.05 -> 71.35 | Helped |
| Combine Saccade and Fixation instead of using either module independently (ablation) | SaFiRe · RefCOCOg and RefCOCO | oIoU | Helped |
| Add Saccade (ablation) | SaFiRe without Saccade · RefCOCOg and RefCOCO | oIoU | Helped |
| Add Fixation (ablation) | SaFiRe without Fixation · RefCOCOg and RefCOCO | oIoU | Helped |
| Replace visual-text concatenation with both Saccade and Fixation operations (ablation) | Naive VSSM concatenation baseline · RefCOCOg and RefCOCO | oIoU | Helped |
| Repeat the text after the image sequence instead of using vanilla [I...I T] ordering (ablation) | SaFiRe Fixation · RefCOCOg and RefCOCO | oIoU | Helped |
| Distribute repeated text throughout image regions instead of repeating it only at the end (ablation) | SaFiRe Fixation · RefCOCOg and RefCOCO | oIoU | Helped |
| Use a 4 × 4 Fixation window rather than other tested window sizes (ablation) | SaFiRe Fixation · RefCOCOg and RefCOCO | oIoU | Helped |
| Apply SaFiRe to a ViT-B backbone rather than the baseline method (ablation) | ViT-B backbone · RefCOCOg | oIoU: +3.78 | Helped |
Limitations
- Struggles with extremely similar visual semantics requiring fine-grained discrimination.
- Fails on unusually long relational chains requiring multi-step reasoning.
- aRefCOCO expressions are LLM-generated, which may introduce bias despite filtering.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg COCO-Stuff aRefCOCO
Benchmarks: RefCOCO RefCOCO+ RefCOCOg aRefCOCO
Metrics: oIoU mIoU Precision@X
Baselines: CRIS VLT LAVT ReLA CGFormer MagNet ReMamber PSALM EEVG PromptRIS OneRef C3VG
Builds on: VMamba Mamba LAVT DiT Swin Transformer
Closest papers by meaning
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.89)
- IteRPrimE: Zero-shot Referring Image Segmentation with Iterative Grad-CAM Refinement and Primary Word Emphasis (AAAI 2025 · similarity 0.88)
- Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentation (CVPR 2025 · similarity 0.87)
- Prompt-Driven Referring Image Segmentation with Instance Contrasting (CVPR 2024 · similarity 0.87)
- RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.87)
- SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.