Research map / Vision-language and multimodal / Human and embodied modeling
ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Ego-IRG task (analyzing, answering, and pixel-grounding egocentric interactions), the Ego-IRGBench dataset built on HOI4D, and the ANNEXE MLLM-based model with a depth-assisted mask branch.
Problem
Existing egocentric interaction understanding methods cannot jointly produce coherent text responses and fine-grained pixel-level masks for user queries, limiting flexibility for downstream tasks.
Approach
A unified model (ANNEXE) built on Mini-GPT4v2 that generates interaction descriptions and query answers as text, plus masks via a mask generation module whose indicator classifier and auxiliary depth-estimation branch (Depth Anything V2 decoder) help separate foreground active objects from background.
Key results
- ANNEXE reaches 36.02 cIoU on the Ego-IRGBench test set versus 23.75 for the best baseline X-Decoder (+12.27).
- Adding the depth estimation branch raises test cIoU from 31.90 to 36.02 and answering CIDEr from 2.317 to 2.590.
- On the test set, analyzing METEOR/CIDEr is 0.563/1.494 for ANNEXE versus 0.179/0.221 for SmallCap.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the depth estimation branch (ablation) | ANNEXE · Ego-IRGBench validation and test | cIoU: 31.72 -> 35.14 (validation); 31.90 -> 36.02 (test) | Helped |
| add the depth estimation branch (ablation) | ANNEXE · Ego-IRGBench validation | METEOR: 0.537 -> 0.563 | Helped |
| add the depth estimation branch (ablation) | ANNEXE · Ego-IRGBench validation | CIDEr: 1.447 -> 1.516 | Helped |
| add the depth estimation branch (ablation) | ANNEXE · Ego-IRGBench validation | METEOR: 0.352 -> 0.363 | Helped |
| add the depth estimation branch (ablation) | ANNEXE · Ego-IRGBench validation | CIDEr: 2.312 -> 2.543 | Helped |
| add the depth estimation branch (ablation) | ANNEXE · Ego-IRGBench test | METEOR: 0.536 -> 0.563 | Helped |
| add the depth estimation branch (ablation) | ANNEXE · Ego-IRGBench test | CIDEr: 1.423 -> 1.494 | Helped |
| add the depth estimation branch (ablation) | ANNEXE · Ego-IRGBench test | METEOR: 0.350 -> 0.365 | Helped |
| add the depth estimation branch (ablation) | ANNEXE · Ego-IRGBench test | CIDEr: 2.317 -> 2.590 | Helped |
| set λdes, λans, and λpxl to 0.5 while keeping λcls and λdep at 1.0 (ablation) | ANNEXE · Ego-IRGBench test | cIoU: 36.02 -> 22.46 | Hurt |
| set all loss weights to 1.0 (ablation) | ANNEXE · Ego-IRGBench test | cIoU: 36.02 -> 23.34 | Hurt |
| use ANNEXE instead of the strongest listed captioning baseline | Ego-IRGBench analyzing · Ego-IRGBench test | CIDEr: 0.221 -> 1.494 | Helped |
Surprising findings
- Lowering the loss weights of the primary tasks relative to the auxiliary classification and depth losses degrades performance substantially (test cIoU drops to about 22-23).
Limitations
- Ego-IRGBench is built on HOI4D with extensive manual annotation, so scale and domain are constrained to that source.
- Baselines are adapted from image captioning and referring segmentation methods, not designed for the unified task.
- No release statement for code, data, or weights is given.
Experimental setup
Datasets: HOI4D Ego-IRGBench
Benchmarks: Ego-IRGBench
Metrics: METEOR CIDEr cIoU
Baselines: ViECap PromptCap SmallCap ConZIC GSVA GRES LISA X-Decoder
Builds on: Mini-GPT4v2 Depth Anything V2 LISA GLaMM
Closest papers by meaning
- EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Grounding (ICML 2026 · similarity 0.91)
- EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views (NeurIPS 2024 · similarity 0.89)
- EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT (NeurIPS 2025 · similarity 0.88)
- The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective (CVPR 2024 · similarity 0.87)
- EgoLife: Towards Egocentric Life Assistant (CVPR 2025 · similarity 0.87)
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.