Research map / Vision-language and multimodal / CLIP and zero-shot
Tune-An-Ellipse: CLIP Has Potential to Find What You Want
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Tune-An-Ellipse, which turns CLIP visual prompting (red ellipses) into a differentiable process and tunes an initial ellipse with an MLP and three losses to zero-shot localize referring expressions without detector proposals.
Problem
Zero-shot referring expression comprehension with CLIP requires precise object proposals from external detectors; visual prompting alone (red circles) is insufficient for localization without them.
Approach
Make the visual prompt (an ellipse rendered via a differentiable Gaussian/sigmoid-approximated distance mask) differentiable, initialize the ellipse from uniformly placed anchor ellipses ranked by CLIP score and Grad-CAM, and train a small MLP to shift ellipse coefficients using a matching loss plus inflation and squeezing Grad-CAM losses.
Key results
- On RefCOCO val/testA/testB with point-anchor ellipse proposals, Tune-An-Ellipse reaches 26.3%/32.3%/19.8%, versus 8.34%/6.56%/10.2% for Red Circle and 7.35%/6.59%/7.69% for ReCLIP.
- With Faster R-CNN proposals, the differentiable prompting variant scores 55.1% on RefCOCO+ val versus 54.7% for the re-implemented Red Circle, a marginal gain.
- Ablation on RefCOCO testA: all three losses together give 32.3% (ensemble Grad-CAM) versus 4.63% for L_sim alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Tune initial anchor ellipses with differentiable visual prompting instead of using them directly | CLIP zero-shot REC · RefCOCO, RefCOCO+, and RefCOCOg | Top-1 accuracy (%): RefCOCO val: 8.34 -> 26.3; testA: 6.56 -> 32.3; testB: 10.2 -> 19.8; RefCOCO+ val: 8.94 -> 27.8; testA: 6.85 -> 32.7; te | Helped |
| Use differentiable visual prompting to score Faster R-CNN proposals | Red Circle with Faster R-CNN proposals · RefCOCO, RefCOCO+, and RefCOCOg | Top-1 accuracy (%): RefCOCO val: 48.6 -> 49.8; testA: 56.2 -> 58.0; testB: 41.7 -> 40.9; RefCOCO+ val: 54.7 -> 55.1; testA: 61.5 -> 62.8; te | Mixed |
| Add Lsim to the Linf and Lsqz objectives (ablation) | Tune-An-Ellipse · RefCOCO testA | Top-1 accuracy (%): 31.2 / 27.2 -> 32.3 / 29.0 | Helped |
| Use an ensemble of two Grad-CAM activation maps (ablation) | Tune-An-Ellipse · RefCOCO testA | Top-1 accuracy (%) | Helped |
| Initialize from the highest matching score rather than the highest average activation (ablation) | Tune-An-Ellipse, 200 tuning steps · RefCOCO testA | Top-1 accuracy (%): 31.6 -> 29.1 | Hurt |
| Select the highest average activation among the Top-10 matching scores (ablation) | Tune-An-Ellipse · RefCOCO testA | Top-1 accuracy (%): 31.8 -> 32.3 | Helped |
| Tune ellipse coefficients for 200 steps rather than not tuning (ablation) | Tune-An-Ellipse · RefCOCO testA | Top-1 accuracy (%): 3.13 -> 32.3 | Helped |
| Vary the number of Top-K candidates, sampled points, and anchor ellipses around the reported setting (ablation) | Tune-An-Ellipse · RefCOCO testA | Top-1 accuracy (%): 31.3; 31.5; 31.6; 31.8; 32.3 | No clear effect |
Surprising findings
- Without any tuning, initial anchor ellipses reach only about 4% accuracy, and Grad-CAM accuracy fluctuates strongly with threshold.
- CLIP often gives higher matching scores to ellipses on background regions or partial inclusions of the target than to the correct full object.
Limitations
- Performance depends on CLIP's visual prompting ability, which fails on background regions, partial inclusions, and sub-word expressions.
- Accuracy remains far below detector-based methods with Faster R-CNN proposals (e.g., 26-32% vs ~45-60%) on RefCOCO-family benchmarks.
- Inflation and squeezing losses rely on Grad-CAM, which is unreliable and can cause over-tuning.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg
Benchmarks: RefCOCO RefCOCO+ RefCOCOg
Metrics: Top-1 accuracy (IoU>0.5) mask accuracy vs threshold
Baselines: Red Circle ReCLIP CPT Pseudo-Q DTWREG
Builds on: Red Circle visual prompting ReCLIP Grad-CAM CLIP CoOp
Closest papers by meaning
- Visual In-Context Prompting (CVPR 2024 · similarity 0.87)
- Interpreting CLIP's Image Representation via Text-Based Decomposition (ICLR 2024 · similarity 0.85)
- un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP (NeurIPS 2025 · similarity 0.85)
- Gradient-based Visual Explanation for Transformer-based CLIP (ICML 2024 · similarity 0.85)
- ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization (AAAI 2024 · similarity 0.85)
- STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.