Research map / Vision-language and multimodal / Human and embodied modeling
LQMFormer: Language-aware Query Mask Transformer for Referring Image Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LQMFormer, a transformer for referring image segmentation that uses Gaussian-enhanced multi-modal fusion, a dynamic query module with a scoring network, and a query-mask margin loss to counter query collapse.
Problem
Transformer-based referring image segmentation suffers from query collapse, where different decoder queries produce identical masks even though RIS needs a single mask, reducing generalization in complex or novel scenes.
Approach
Combines Fourier-domain Gaussian-filtered vision-language fusion inside the Swin backbone with language-conditioned query generation and a learned scoring network that selects relevant queries, plus an auxiliary loss that pushes apart mask embeddings of different queries.
Key results
- On GRES validation, LQMFormer reaches 64.98 oIoU and 70.94 gIoU, versus 62.42 oIoU and 63.60 gIoU for ReLA.
- No-target accuracy on GRES validation is 67.47%, reported as about 11 points above existing methods.
- On RefCOCO, LQMFormer gets the highest oIoU on val (74.16), test A (76.82) and test B (71.04) among the compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LQMFormer instead of ReLA | Referring image segmentation · RefCOCO test B | oIoU: 70.18 -> 71.04 | Helped |
| Use LQMFormer instead of ReLA | Referring image segmentation · RefCOCO+ test A | oIoU: 71.02 -> 71.84 | Helped |
| Use LQMFormer instead of ReLA | Referring image segmentation · RefCOCO+ val | oIoU: 66.04 -> 65.91 | Hurt |
| Use LQMFormer instead of ReLA | Referring image segmentation · G-Ref val(U) | oIoU: 65.00 -> 64.73 | Hurt |
| Use LQMFormer instead of ReLA | GRES · GRES val | gIoU: 63.60 -> 70.94 | Helped |
| Add score-based learning (ablation) | LQMFormer · GRES val | oIoU: 62.75 -> 64.98 | Helped |
| Add the scoring module (ablation) | LQMFormer · GRES val | oIoU: 62.02 -> 64.98 | Helped |
| Add QM-loss (ablation) | LQMFormer · GRES val | oIoU: 64.19 -> 64.98 | Helped |
| Use 100 queries instead of 20 (ablation) | LQMFormer · GRES val | oIoU: 57.22 -> 64.98 | Helped |
| Use 100 queries instead of 50 (ablation) | LQMFormer · GRES val | oIoU: 62.48 -> 64.98 | Helped |
| Add the scoring module (ablation) | LQMFormer with 100 queries · GRES val | oIoU: 62.02 -> 64.98 | Helped |
| Select the top 20% of queries rather than the top 50% (ablation) | LQMFormer · GRES val | oIoU: 63.85 -> 64.98 | Helped |
Surprising findings
- Removing the scoring module (keeping 100 queries) drops GRES oIoU by 2.96 points, while increasing the top-p% query ratio to 50% or 100% degrades performance due to query collapse.
Limitations
- Gains on classic RefCOCO+ and RefCOCOg are smaller and mixed, with LQMFormer not best on several splits.
- Hyperparameters such as query count and top-p% ratio are tuned on GRES, and the paper does not report compute cost.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg GRES
Benchmarks: RefCOCO RefCOCO+ RefCOCOg GRES
Metrics: oIoU Precision@X gIoU N-acc. T-acc.
Baselines: MCN VLT ReSTR CRIS LAVT ReLA MattNet LTS
Builds on: LAVT Mask2Former ReLA BERT Swin Transformer
Closest papers by meaning
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.93)
- OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling (NeurIPS 2024 · similarity 0.89)
- Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentation (CVPR 2025 · similarity 0.89)
- Prompt-Driven Referring Image Segmentation with Instance Contrasting (CVPR 2024 · similarity 0.88)
- GSVA: Generalized Segmentation via Multimodal Large Language Models (CVPR 2024 · similarity 0.87)
- Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.