Research map / Vision-language and multimodal / Human and embodied modeling
Weakly Supervised Multimodal Affordance Grounding for Egocentric Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WSMA, a weakly supervised multimodal framework that transfers affordance knowledge from exocentric images and affordance text to egocentric images for affordance grounding using only image-level labels.
Problem
Pixel-level affordance annotation is costly and error-prone, and prior affordance grounding methods ignore textual action cues and rely mainly on exocentric images.
Approach
Combines a local HOI-Transfer Module (CAM-based heatmap distillation and action-correlation loss from exocentric to egocentric views) with a Pixel-Text Fusion Module (CLIP text prompts fused into egocentric pixel features), trained with image-level labels only.
Key results
- On AGD20K-Unseen, WSMA achieves KLD 1.335, SIM 0.382, NSS 1.220, improving over LOCATE (1.405, 0.372, 1.157).
- On AGD20K-Seen, WSMA achieves KLD 1.176, SIM 0.416, NSS 1.247, versus LOCATE's 1.226, 0.401, 1.177.
- On HICO-IIF, WSMA achieves KLD 1.465, SIM 0.358, NSS 1.012, outperforming LOCATE (1.593, 0.327, 0.966).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the HOI-Transfer Module and Pixel-Text Fusion Module (ablation) | WSMA · ADE20K-Unseen | KLD: 1.707 -> 1.335 | Helped |
| add the HOI-Transfer Module and Pixel-Text Fusion Module (ablation) | WSMA · ADE20K-Seen | KLD: 1.430 -> 1.176 | Helped |
| add the HOI-Transfer Module and Pixel-Text Fusion Module (ablation) | WSMA · HICO-IIF | KLD: 1.860 -> 1.465 | Helped |
| add the Pixel-Text Fusion Module (ablation) | WSMA · ADE20K-Unseen | KLD: 1.471 -> 1.335 | Helped |
| add the Pixel-Text Fusion Module (ablation) | WSMA · ADE20K-Seen | KLD: 1.297 -> 1.176 | Helped |
| add the Pixel-Text Fusion Module (ablation) | WSMA · HICO-IIF | KLD: 1.785 -> 1.465 | Helped |
| add the HOI-Transfer Module (ablation) | WSMA · ADE20K-Unseen | KLD: 1.531 -> 1.335 | Helped |
| add the HOI-Transfer Module (ablation) | WSMA · ADE20K-Seen | KLD: 1.277 -> 1.176 | Helped |
| add the HOI-Transfer Module (ablation) | WSMA · HICO-IIF | KLD: 1.690 -> 1.465 | Helped |
| add Ld loss (ablation) | WSMA HOI-Transfer Module · ADE20K-Unseen | KLD: 1.531 -> 1.335 | Helped |
| add Ld loss (ablation) | WSMA HOI-Transfer Module · ADE20K-Seen | KLD: 1.277 -> 1.176 | Helped |
| add Ld loss (ablation) | WSMA HOI-Transfer Module · HICO-IIF | KLD: 1.690 -> 1.465 | Helped |
Surprising findings
- Replacing the classification-score-based action correlation loss with heatmap-based correlation (Ll_rela) gave consistent gains, while adding the classification-based loss alongside Ld did not improve all metrics.
Limitations
- Relies on existing public datasets that lack complex interaction images, e.g., multiple objects of different categories sharing one affordance label.
- Evaluated only on AGD20K and a newly assembled HICO-IIF subset.
Experimental setup
Datasets: AGD20K HICO-IIF HICO-DET IIT-AFF ImageNet
Benchmarks: AGD20K-Seen AGD20K-Unseen HICO-IIF
Metrics: KLD SIM NSS
Baselines: Cross-view-AG Cross-view-AG+ LOCATE Hotspots SPA EIL TS-CAM
Builds on: Cross-view-AG LOCATE CoOp Affordance Invariance Mining Module CAM
Closest papers by meaning
- Weakly-Supervised Affordance Grounding Guided by Part-Level Semantic Priors (ICLR 2025 · similarity 0.91)
- Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding (AAAI 2025 · similarity 0.90)
- Reasoning Mamba: Hypergraph-Guided Region Relation Calculating for Weakly Supervised Affordance Grounding (CVPR 2025 · similarity 0.89)
- Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions (CVPR 2025 · similarity 0.89)
- SeqAfford: Sequential 3D Affordance Reasoning via Multimodal Large Language Model (CVPR 2025 · similarity 0.88)
- IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.