Research map / Vision-language and multimodal / CLIP and zero-shot
Densely Connected Parameter-Efficient Tuning for Referring Image Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DETRIS, a parameter-efficient tuning framework that adapts a frozen DINOv2 backbone and CLIP text encoder for referring image segmentation using Dense Aligner adapters and text adapters.
Problem
Parameter-efficient tuning for referring image segmentation has mainly been studied with aligned encoders such as CLIP, and performs poorly with misaligned vision encoders (e.g., DINO) that lack visual-text pretraining.
Approach
Dense Aligner adapters inserted in parallel to MLP layers of a frozen DINOv2 ViT, combining a dense multi-scale convolution mixture (D-MoC) with cross-modal multi-head cross-attention to text, plus D-MoC-style 1D text adapters on the frozen CLIP text encoder.
Key results
- DETRIS-L reaches 72.2 average IoU across RefCOCO/RefCOCO+/G-Ref, outperforming prior methods, and DETRIS-B reaches 70.4 with 0.9% to 1.8% backbone parameters updated.
- With DINO-B backbone on RefCOCO, DETRIS-B (default setting, 2.71M params) reaches 75.9 average versus ETRIS 74.6 with 1.38M params.
- Ablation: Dense Aligner alone gives 75.0 and Text Adapter alone gives 75.4 average, versus 74.6 baseline; both combined give 75.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DETRIS-L instead of RISCLIP-B | Referring image segmentation · RefCOCO/RefCOCO+/G-Ref (no extra data) | Avg IoU: 70.6 -> 72.2 | Helped |
| train DETRIS-L on mixed RefCOCO/RefCOCO+/G-Ref datasets | DETRIS-L · Mixed RefCOCO datasets | Avg IoU: 74.4 -> 77.2 | Helped |
| use DETRIS-B default setting instead of ETRIS | DINO-B on RefCOCO · RefCOCO | Avg IoU: 74.6 -> 75.9 | Helped |
| use full fine-tuning instead of freezing the DINO-B backbone | DINO-B on RefCOCO · RefCOCO | Avg IoU: 74.7 -> 64.9 | Hurt |
| add Dense Aligner (ablation) | DETRIS-B without adapters · RefCOCO | Avg IoU: 74.6 -> 75.0 | Helped |
| add Text Adapter (ablation) | DETRIS-B without adapters · RefCOCO | Avg IoU: 74.6 -> 75.4 | Helped |
| add Text Adapter to a model with Dense Aligner (ablation) | DETRIS-B with Dense Aligner only · RefCOCO | Avg IoU: 75.0 -> 75.9 | Helped |
| add Dense Aligner to a model with Text Adapter only (ablation) | DETRIS-B with Text Adapter only · RefCOCO | Avg IoU: 75.4 -> 75.9 | Helped |
| reduce Dense Aligners from 6 to 3 (ablation) | DETRIS-B · RefCOCO | Avg IoU: 75.9 -> 75.5 | Hurt |
| increase visual dimension from 64 to 128 while keeping text dimension 64 (ablation) | DETRIS-B · RefCOCO | val IoU: 75.8 -> 76.0 | Helped |
| increase text dimension from 64 to 128 while visual dimension is 128 (ablation) | DETRIS-B · RefCOCO | val IoU: 76.0 -> 76.6 | Helped |
| increase visual dimension from 128 to 256 while text dimension is 128 (ablation) | DETRIS-B · RefCOCO | val IoU: 76.6 -> 76.4 | Hurt |
Surprising findings
- Fully fine-tuning the DINO backbone degrades RIS performance (64.9 average vs 74.7 for a fixed backbone), indicating misaligned encoders are hard to fully fine-tune.
- The Text Adapter alone yields a larger gain (75.4) than the Dense Aligner alone (75.0).
Limitations
- Experiments limited to referring image segmentation; not validated on other multimodal segmentation tasks.
- Attempts to inject visual information into the text adapter yielded minimal gains or added many parameters.
- Open-vocabulary zero-shot RIS is not explored.
Experimental setup
Datasets: RefCOCO RefCOCO+ G-Ref
Benchmarks: RefCOCO RefCOCO+ G-Ref
Metrics: mIoU overall IoU
Baselines: ETRIS BarLeRIa CRIS LAVT ReLA CGFormer LISA-7B MagNet ReMamber RISCLIP-B PolyFormer-L UNINEXT-L
Builds on: ETRIS CRIS DINOv2 LoRA Adapter
Closest papers by meaning
- TTE: Two Tokens Are Enough to Improve Parameter-Efficient Tuning (AAAI 2025 · similarity 0.87)
- Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model (CVPR 2024 · similarity 0.86)
- Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation (CVPR 2025 · similarity 0.85)
- Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition (CVPR 2025 · similarity 0.85)
- Gradient-based Parameter Selection for Efficient Fine-Tuning (CVPR 2024 · similarity 0.85)
- Minimal Interaction Seperated Tuning: A New Paradigm for Visual Adaptation (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.