Research map / Vision-language and multimodal / Human and embodied modeling
Revisiting Counterfactual Problems in Referring Expression Comprehension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds fine-grained counterfactual referring-expression datasets (C-RefCOCO/+/g) by BERT-based attribute word replacement and proposes a one-stage C-REC model with dual-branch attentive fusion and contrastive learning that jointly predicts a box and a counterfactual label.
Problem
Referring expression comprehension assumes the target always exists; counterfactual queries whose referent is absent from the image (especially with fine-grained attribute mismatches) produce wrong boxes and are not handled.
Approach
A labor-free counterfactual sample generation method that masks dependency-parsed attribute words and re-ranks BERT predictions to produce fine-grained negative queries, plus a one-stage model with a dual-branch attentive fusion (image-text and image-attribute) and contrastive learning using these negatives.
Key results
- On C-RefCOCO/+/g the binary counterfactual classifier reaches about 90% Acc-Cls (e.g., 92.31% testA on C-RefCOCO), surpassing confidence-threshold baselines by up to 15.73%.
- Acc-Cf on C-RefCOCO testB rises from 37.62% baseline to 82.44% with attribute features, contrastive loss and counterfactual training combined (ablation).
- On standard RefCOCO/+/g the one-stage model reports 82.77/86.35/77.13 Acc-Box on RefCOCO val/testA/testB, competitive without VLP pretraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed C-REC framework rather than the strongest reported baseline | REC localization · RefCOCO/+/g | Acc-Box (%): 82.77 vs 92.40; 86.35 vs 94.51; 77.13 vs 88.42; 72.29 vs 86.02; 78.24 vs 90.17; 63.47 vs 79.22; 73.33 vs 85.89; 74.11 vs | Mixed |
| use a binary classifier instead of confidence-score thresholding or random choice | C-REC counterfactual detection · C-RefCOCO/+/g | Acc-Cls (%): 93.05 vs 86.25; 92.31 vs 86.05; 91.69 vs 82.67; 91.98 vs 87.88; 91.09 vs 87.62; 89.21 vs 84.15; 89.45 vs 73.72; 89.08 vs | Helped |
| add attribute features (ablation) | C-REC model · C-RefCOCO testB | Acc-Box (%): 75.23 -> 78.05 | Helped |
| add contrastive loss (ablation) | C-REC model · C-RefCOCO testB | Acc-Box (%): 75.23 -> 77.96 | Helped |
| add counterfactual training (ablation) | C-REC model · C-RefCOCO testB | Acc-Cls (%): 50.00 -> 86.74 | Helped |
| combine attribute features, contrastive loss, and counterfactual training (ablation) | C-REC model · C-RefCOCO testB | Acc-Cf (%): 37.62 -> 82.44 | Helped |
| replace image-text-only fusion with serial attribute-to-image-text fusion (ablation) | C-REC model · C-RefCOCO | Acc-Cf (%): 83.72 -> 82.04 | Hurt |
| replace image-text-only fusion with serial text-to-image-attribute fusion (ablation) | C-REC model · C-RefCOCO | Acc-Cf (%): 83.72 -> 83.17 | Hurt |
| use parallel fusion of image-attribute and image-text features (ablation) | C-REC model · C-RefCOCO | Acc-Cf (%): 83.72 -> 86.32 | Helped |
| set contrastive-loss temperature τ to 0.05, 0.1, or 0.5 instead of 0.2 (ablation) | C-REC model · C-RefCOCO | Acc-Cf (%): 86.32 -> 85.43; 86.32 -> 85.47; 86.32 -> 84.83 | Hurt |
| vary DAF weight α for counterfactual detection and regression (ablation) | C-REC model · C-RefCOCO | Acc-Cf (%) | Mixed |
Surprising findings
- Parallel attribute-guided fusion outperforms serial fusion orderings, which slightly decrease performance relative to the baseline.
Limitations
- Counterfactual samples are text-only; no counterfactual images are generated.
- Thresholding confidence scores generalizes poorly to long, complex queries (C-RefCOCOg).
- Model struggles with multi-attribute queries, e.g., missing the size attribute 'small' and over-attending to absolute location.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg MS-COCO
Benchmarks: RefCOCO RefCOCO+ RefCOCOg C-RefCOCO C-RefCOCO+ C-RefCOCOg
Metrics: Acc-Box (IoU@0.5) Acc-Cls Acc-Cf
Baselines: MDETR OFA m-PLUG FAOA ReSC MCN RealGIN LG-FPN PFOS VGTR TransVG SimREC
Builds on: SimREC RealGIN InfoNCE CSPDarkNet
Closest papers by meaning
- Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks (ICLR 2026 · similarity 0.89)
- Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point (AAAI 2024 · similarity 0.88)
- Exploring Contextual Attribute Density in Referring Expression Counting (CVPR 2025 · similarity 0.88)
- RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.88)
- ScanFormer: Referring Expression Comprehension by Iteratively Scanning (CVPR 2024 · similarity 0.87)
- DiffusionREC: Diffusion Model with Adaptive Condition for Referring Expression Comprehension (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.