Research map / Vision-language and multimodal / CLIP and zero-shot
Alpha-CLIP: A CLIP Model Focusing on Wherever You Want
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds an alpha (region mask) input channel to the CLIP image encoder, fine-tuned on millions of auto-generated RGBA region-text pairs, so CLIP can focus on user-specified regions without altering the RGB image, and plugs into downstream recognition, MLLM, and 2D/3D generation systems.
Problem
CLIP encodes the whole image, but many tasks need region-focused features for user-specified points, masks, or boxes; prior methods crop, mask, or draw on the image, losing context or altering image content.
Approach
Add a parallel alpha convolution to the CLIP image encoder so it accepts an RGBA input, initialized to zero so behavior matches CLIP at start, and train it on automatically generated RGBA region-text pairs mixed with full-alpha image-text pairs.
Key results
- On ImageNet-S zero-shot classification with ViT-L/14, Alpha-CLIP reaches 77.41% top-1 vs 73.48% for original CLIP when given ground-truth masks.
- On zero-shot REC, Alpha-CLIP surpasses ReCLIP and Red Circle by averages of 6.8% and 3.0% accuracy.
- On OV-LVIS, MaskImageNet with Alpha-CLIP reaches 28.6 novel mAP vs 24.6 for Detic-ImageNet, using 460K images vs 1.2M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide a foreground mask through the alpha channel rather than use the original CLIP region input | CLIP on ImageNet-S · ImageNet-S | Top-1 accuracy: 77.04 -> 77.41 | Helped |
| Replace CLIP with Alpha-CLIP in the zero-shot REC approach | Zero-shot REC · RefCOCO, RefCOCO+, RefCOCOg | Top-1 accuracy | Mixed |
| Use MaskImageNet with Alpha-CLIP for OVD pseudo-labeling | OVD · OV-LVIS | mAPnovel: 24.6 -> 28.6 | Helped |
| Replace the LLaVA vision CLIP with Alpha-CLIP | LLaVA region-level captioning · RefCOCOg | CIDEr: 105.0 -> 109.2 | Helped |
| Use a whole-image alpha map instead of a region mask (ablation) | Alpha-CLIP · ImageNet-S | Top-1 accuracy: 77.41 -> 73.37 | Hurt |
| Use a rectangular-box alpha map instead of a mask (ablation) | Alpha-CLIP · ImageNet-S | Top-1 accuracy: 77.41 -> 75.62 | Hurt |
Surprising findings
- Alpha-CLIP with an all-ones alpha map performs on par with original CLIP (73.37 vs 73.48 top-1), so the alpha input does not hurt full-image recognition.
- Cropping to the object box was found to be worse than feeding the full image with an alpha map for zero-shot REC.
Limitations
- Requires a foreground mask, box, or stroke at inference to realize its benefit; without it, performance matches original CLIP.
- Relies on pseudo-labels from SAM and BLIP-2 for training data quality.
- Some downstream results (e.g., 3D, MLLM) are qualitative or limited to a few benchmarks.
Experimental setup
Datasets: GRIT-20m ImageNet ImageNet-S RefCOCO RefCOCO+ RefCOCOg Visual Genome LVIS OV-LVIS
Benchmarks: ImageNet-S RefCOCO RefCOCO+ RefCOCOg OV-LVIS Visual Genome
Metrics: top-1 accuracy top-5 accuracy mAP METEOR CIDEr
Baselines: Original CLIP MaskAdaptedCLIP Red Circle MaskCLIP CPT ReCLIP Detic GLaMM GPT4RoI
Builds on: CLIP SAM GRIT BLIP-2 LLaVA-1.5
Closest papers by meaning
- Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification (CVPR 2024 · similarity 0.85)
- Interpreting CLIP's Image Representation via Text-Based Decomposition (ICLR 2024 · similarity 0.84)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.84)
- un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP (NeurIPS 2025 · similarity 0.83)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.81)
- Tune-An-Ellipse: CLIP Has Potential to Find What You Want (CVPR 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.