Research map / Detection and segmentation / Weakly supervised detection and segmentation
Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Replaces CAM-based weakly supervised affordance grounding with a pseudo-fully supervised pipeline, generating heatmap labels from VLpart+SAM part segmentation via an affordance-to-part mapping, then refining labels, aligning with exocentric features, and adding a part-reasoning module.
Problem
Weakly supervised affordance grounding (WSAG): locating affordance regions on static object images using only image-level labels and exocentric interaction images, where CAM-based methods are inaccurate for actions and functions.
Approach
Convert affordance queries to object-part names via a mapping, use an off-the-shelf part detector (VLpart) plus SAM to produce pseudo heatmap labels for egocentric images, then train supervised with KL loss, enhanced by SAM-based label refinement guided by exocentric occlusion cues, mask-pooled cross-view alignment, and a noun/part reasoning module.
Key results
- On AGD20K unseen split, the full model reaches KLD 1.153, SIM 0.437, NSS 1.418, improving over the best prior WSMA (1.335, 0.382, 1.220).
- On the seen split, the full model reaches KLD 0.890, SIM 0.510, NSS 1.547, versus WSMA's 1.176, 0.416, 1.247.
- The pseudo-supervised baseline alone already beats all CAM-based prior methods on both splits (e.g., seen KLD 0.938 vs 1.176).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace CAM-based weak supervision with part-level pseudo labels and supervised KL training | WSAG on AGD20K · AGD20K seen split | KLD: 1.176 -> 0.938 | Helped |
| Replace CAM-based weak supervision with part-level pseudo labels and supervised KL training | WSAG on AGD20K · AGD20K unseen split | KLD: 1.335 -> 1.256 | Helped |
| Add label refinement, cross-view alignment, and reasoning to the pseudo-label baseline | Ours-baseline · AGD20K seen split | KLD: 0.938 -> 0.890 | Helped |
| Add label refinement, cross-view alignment, and reasoning to the pseudo-label baseline | Ours-baseline · AGD20K unseen split | KLD: 1.256 -> 1.153 | Helped |
| Add label refinement to Ours-baseline (ablation) | Ours-baseline · AGD20K seen split | KLD: 0.938 -> 0.916 | Helped |
| Add label refinement to Ours-baseline (ablation) | Ours-baseline · AGD20K unseen split | KLD: 1.256 -> 1.246 | No clear effect |
| Add cross-view feature alignment to Ours-baseline (ablation) | Ours-baseline · AGD20K seen split | KLD: 0.938 -> 0.924 | Helped |
| Add cross-view feature alignment to Ours-baseline (ablation) | Ours-baseline · AGD20K unseen split | KLD: 1.256 -> 1.176 | Helped |
| Add the reasoning module to Ours-baseline (ablation) | Ours-baseline · AGD20K seen split | KLD: 0.938 -> 0.943 | Hurt |
| Add the reasoning module to Ours-baseline (ablation) | Ours-baseline · AGD20K unseen split | KLD: 1.256 -> 1.242 | Helped |
Surprising findings
- Off-the-shelf MLLM and open-vocabulary detector affordance queries (e.g., VLpart with affordance text, MiniGPT-v2, Qwen-VL) performed poorly, and part-name queries were needed instead.
- The reasoning module gave little benefit on the seen split but helped on the unseen split.
Limitations
- Pseudo labels from VLpart+SAM remain imperfect for some part names and affordances without well-defined parts.
- The affordance-to-part mapping P is manually defined (around 100 combinations), limiting scalability.
- Generalization to unseen objects remains weaker than seen-split performance.
Experimental setup
Datasets: AGD20K
Benchmarks: AGD20K seen split AGD20K unseen split
Metrics: KLD SIM NSS
Baselines: Cross-View-AG Cross-View-AG+ LOCATE WSMA Strategies WorldAfford
Builds on: CLIP SAM VLpart Cross-View-AG LOCATE
Closest papers by meaning
- Weakly Supervised Multimodal Affordance Grounding for Egocentric Images (AAAI 2024 · similarity 0.91)
- Reasoning Mamba: Hypergraph-Guided Region Relation Calculating for Weakly Supervised Affordance Grounding (CVPR 2025 · similarity 0.91)
- Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding (AAAI 2025 · similarity 0.89)
- Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions (CVPR 2025 · similarity 0.88)
- SeqAfford: Sequential 3D Affordance Reasoning via Multimodal Large Language Model (CVPR 2025 · similarity 0.88)
- 3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.