Research map / Vision-language and multimodal / CLIP and zero-shot
Text Augmented Correlation Transformer For Few-shot Classification & Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TA-CST, a multi-modal few-shot classification and segmentation model that uses CLIP text (class labels) and image support via dual correlation transformers, a feature enrichment module, and zero-shot distillation.
Problem
Integrative few-shot classification and segmentation (FS-CS) relies on visual support only, which struggles with ambiguous boundaries and overlapping classes; text support (class labels) is unexplored for FS-CS.
Approach
Uses frozen CLIP ViT-B/16 features to build vision-only and vision-text correlation maps, processed by two correlation transformers, with a granular key-value atrous feature enrichment module and zero-shot distillation from CLIP positive/negative prompts; supports text-only and text-augmented predictions.
Key results
- Text-augmented TA-CST improves 1-way 1-shot classification/segmentation by +3.7/+6.6% on COCO-20i over CST (CLIP-B/16) per the abstract.
- Text-only support alone achieves 90.3% classification and 49.3% mIoU on COCO-20i 1-way 1-shot, competitive with vision-only methods.
- In weakly supervised Pascal-5i, text-only and text-augmented variants surpass CST (CLIP-B/16) with text support, per Table 3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine text and image support predictions instead of using CST (CLIP-B/16) | TA-CST · COCO-20i 1-way-1-shot | classification exact ratio (%): 88.4 -> 92.1 | Helped |
| combine text and image support predictions instead of using CST (CLIP-B/16) | TA-CST · COCO-20i 1-way-1-shot | segmentation mIoU (%): 46.3 -> 52.9 | Helped |
| combine text and image support predictions instead of using CST (CLIP-B/16) | TA-CST · Pascal-5i 1-way-1-shot | classification exact ratio (%): 89.0 -> 91.2 | Helped |
| combine text and image support predictions instead of CST (CLIP-B/16) | TA-CST weakly supervised · Pascal-5i weakly supervised 1-way-1-shot | segmentation mIoU (%): 38.5 -> 42.6 | Helped |
| increase text augmentation from 0% to 100% (ablation) | TA-CST · Pascal-5i 1-way-1-shot | classification exact ratio (%): 89.8 -> 91.2 | Helped |
| increase text augmentation from 0% to 100% (ablation) | TA-CST · Pascal-5i 1-way-1-shot | segmentation mIoU (%): 62.4 -> 66.2 | Helped |
| combine text and image support instead of image-only support (ablation) | TA-CST · Pascal-5i 1-way-1-shot | segmentation mIoU (%): 62.4 -> 66.2 | Helped |
| combine text and image support instead of text-only support (ablation) | TA-CST · Pascal-5i 1-way-1-shot | segmentation mIoU (%): 59.3 -> 66.2 | Helped |
| add the feature enrichment module with granular KV (ablation) | TA-CST text-only · Pascal-5i 1-way-1-shot | segmentation mIoU (%): 53.7 -> 59.3 | Helped |
| add zero-shot augmented class predictions and cross-entropy loss (ablation) | TA-CST text-only · Pascal-5i | classification exact ratio (%): 85.3 -> 90.2 | Helped |
| add zero-shot augmented class predictions and cross-entropy loss (ablation) | TA-CST text-only · COCO-20i | classification exact ratio (%): 90.1 -> 90.3 | Helped |
| use zero-shot augmentation without cross-entropy loss (ablation) | TA-CST text-only · Pascal-5i | classification exact ratio (%): 85.7 -> 84.0 | Hurt |
Surprising findings
- Text-only support on Pascal-5i trails image-based predictions, but is comparable on COCO-20i.
- Zero-shot distillation loss was excluded in the text-augmented setting because it gave no gain there.
Limitations
- Text-only classification lags on smaller datasets such as Pascal-5i without zero-shot distillation.
- Text-only predictions tend to over-segment and misclassify object boundaries.
Experimental setup
Datasets: COCO-20i Pascal-5i
Benchmarks: COCO-20i Pascal-5i
Metrics: classification exact ratio segmentation mIoU
Baselines: CST CST (CLIP-B/16) ASNet HSNet PFENet PANet GPT-4Vision CLIP-B/16 zero-shot
Builds on: CST ASNet CLIP
Closest papers by meaning
- Label-Efficient Few-Shot Semantic Segmentation with Unsupervised Meta-Training (AAAI 2024 · similarity 0.90)
- Adaptive FSS: A Novel Few-Shot Segmentation Framework via Prototype Enhancement (AAAI 2024 · similarity 0.88)
- VT-FSL: Bridging Vision and Text with LLMs for Few-Shot Learning (NeurIPS 2025 · similarity 0.88)
- Unlocking the Potential of Pre-trained Vision Transformers for Few-Shot Semantic Segmentation through Relationship Descriptors (CVPR 2024 · similarity 0.88)
- Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation (ICLR 2025 · similarity 0.88)
- A Simple Framework for Open-Vocabulary Zero-Shot Segmentation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.