Research map / Vision-language and multimodal / CLIP and zero-shot
One-Shot Open Affordance Learning with Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces one-shot open affordance learning (OOAL), training a DINOv2+CLIP-text affordance segmentation model with one example per base object category and evaluating zero-shot on novel objects and affordances.
Problem
Affordance segmentation normally needs large annotated datasets and works only over a closed affordance vocabulary; vision-language models fail to recognize fine-grained affordances and parts. The paper targets learning affordances from one example per base object and generalizing to unseen objects
Approach
Analyze foundation models for affordance awareness, then pick DINOv2 for part-aware visual features and align them to frozen CLIP text embeddings via three modules: CoOp-style text prompt learning, multi-layer feature fusion, and a CLS-token-guided transformer decoder with masked cross-attention.
Key results
- On AGD20K unseen split, the method reaches KLD 1.070, SIM 0.461, NSS 1.503, beating the best WSAG method LOCATE (KLD 1.405, SIM 0.372, NSS 1.157) using about 0.2% of the training data.
- On UMD, the method reaches hIoU 66.4 (seen 74.6, unseen 59.7), comparable to fully supervised SegFormer/PSPNet trained on the full set (hIoU 65.0 and 66.0).
- Ablation on AGD20K shows DINOv2 outperforms CLIP and DeiT III as a visual encoder, and each proposed module improves over the DINOv2 baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed DINOv2 + CLIP framework instead of ZegCLIP | OOAL affordance segmentation · AGD20K seen | NSS: 1.001 -> 1.745 | Helped |
| use the proposed DINOv2 + CLIP framework instead of ZegCLIP | OOAL affordance segmentation · AGD20K unseen | NSS: 1.042 -> 1.503 | Helped |
| use the proposed method instead of SegFormer in the OOAL setting | UMD affordance segmentation · UMD | hIoU: 57.3 -> 66.4 | Helped |
| replace DINOv2 with CLIP as the visual encoder (ablation) | OOAL baseline · AGD20K seen | NSS: 1.297 -> 1.107 | Hurt |
| replace DINOv2 with DeiT III as the visual encoder (ablation) | OOAL baseline · AGD20K seen | NSS: 1.297 -> 1.140 | Hurt |
| add text prompt learning (ablation) | DINOv2 baseline · AGD20K seen | NSS: 1.297 -> 1.422 | Helped |
| add multi-layer feature fusion (ablation) | DINOv2 baseline with text prompt learning · AGD20K seen | NSS: 1.422 -> 1.622 | Helped |
| add the transformer decoder without the CLS-guided mask (ablation) | DINOv2 baseline with text prompt learning and multi-layer feature fusion · AGD20K seen | NSS: 1.622 -> 1.738 | Helped |
| add the CLS-guided mask to the transformer decoder (ablation) | OOAL model with transformer decoder · AGD20K unseen | NSS: 1.408 -> 1.503 | Helped |
| add the CLS-guided mask to the transformer decoder (ablation) | OOAL model with transformer decoder · AGD20K seen | SIM: 0.578 -> 0.577 | No clear effect |
Surprising findings
- Vanilla CLIP (MaskCLIP) performs very poorly on affordance grounding, with near-zero NSS on AGD20K, and fully supervised segmentation models degrade by about 10% when trained one-shot.
- The transformer decoder alone improves seen performance but hurts unseen performance; adding the CLS-guided mask recovers unseen gains.
Limitations
- Evaluated only on two affordance segmentation datasets, with dense prediction as the sole task.
- Vision-only DINOv2 backbone with text alignment depends on CLIP text encoder, and unseen-affordance results are only shown qualitatively.
- AGD20K training required manual annotation of 50 egocentric images per object category.
Experimental setup
Datasets: AGD20K UMD part affordance dataset
Benchmarks: AGD20K UMD part affordance
Metrics: KLD SIM NSS mIoU hIoU
Baselines: LOCATE Cross-view-AG Cross-view-AG+ Hotspots MaskCLIP SAN ZegCLIP PSPNet DeepLabV3+ SegFormer
Builds on: CLIP DINOv2 CoOp MaskCLIP MMSegmentation
Closest papers by meaning
- MaskPrompt: Open-Vocabulary Affordance Segmentation with Object Shape Mask Prompts (AAAI 2025 · similarity 0.89)
- 3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds (ICLR 2025 · similarity 0.88)
- Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors (ICLR 2025 · similarity 0.87)
- SeqAfford: Sequential 3D Affordance Reasoning via Multimodal Large Language Model (CVPR 2025 · similarity 0.86)
- Weakly Supervised Multimodal Affordance Grounding for Egocentric Images (AAAI 2024 · similarity 0.86)
- Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.