Research map / Vision-language and multimodal / Human and embodied modeling
Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WAGIBench, a scripted egocentric multimodal benchmark (video, audio, digital, longitudinal context) for inferring user goals for assistive wearable agents, evaluated with MCQ and LLM-judge generative metrics.
Problem
Wearable assistant agents require users to fully specify queries; there is no ecologically valid benchmark with ground-truth goals that jointly uses egocentric video, audio, digital app state, and longitudinal history to infer user goals.
Approach
A scripted data collection with Aria glasses recordings, LLM-generated digital app states, and Socratic (caption-based) longitudinal histories, paired with both sentenceBERT-sampled MCQs and an LLM judge validated against human raters.
Key results
- Human MCQ accuracy reaches 93% (similar distractors) and 97% (dissimilar), while the best model (Qwen-based) gets about 84% on similar MCQs.
- LLM judge conditioned on reference goal and script cues reaches 76.8% pairwise alignment with humans, comparable to human-human agreement of 75.2%.
- Adding audio context improves performance by as much as 35% on MCQ and 30% on generative evaluation over vision-only context on the VA subset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| condition the judge on both reference goal and script cues | LLM Judge · WAGIBench, generative meta-evaluation | pairwise comparison accuracy: 76.8% | Helped |
| use Socratic context instead of reference and script cues (ablation) | LLM Judge · WAGIBench, generative meta-evaluation | pairwise comparison accuracy: 63.0% | Hurt |
| map generated goals onto fixed MCQ options (ablation) | generative evaluation · WAGIBench, generative meta-evaluation | pairwise comparison accuracy: 67.8% | Hurt |
| replace LLM judging with sentenceBERT similarity (ablation) | generative evaluation · WAGIBench, generative meta-evaluation | pairwise comparison accuracy: 59.5% | Hurt |
| increase model size within a model family | VLMs · WAGIBench, discriminative and generative tasks | performance | Helped |
| add audio transcription to vision input (ablation) | VLMs · WAGIBench, S_VA | performance: as large as 35% on the MCQ task and 30% on the generative evaluation | Helped |
| add relevant digital app states to vision input (ablation) | VLMs · WAGIBench, S_VD | performance: as much as 12% | Helped |
| use only relevant digital app states instead of full digital context (ablation) | VLMs · WAGIBench, S_VD | performance: as much as 12% | Helped |
| use only positive-support history instead of full longitudinal history (ablation) | VLMs · WAGIBench, S_VL | performance: at most 5.6% | Helped |
| provide all modalities, including distracting digital and longitudinal context (ablation) | small and medium VLMs · WAGIBench, full benchmark | performance | Hurt |
Surprising findings
- Digital and longitudinal contexts give less benefit than audio, attributed to low signal-to-noise ratio; high-signal versions (D*, L*) help more.
- Socratic (caption-based) context underperforms at 63.0% alignment, and SBERT similarity performs poorly at 59.5%.
Limitations
- Human predictability validated only for vision and audio, not digital or longitudinal contexts.
- Assumes user-initiated interactions; no negative samples for proactive when-to-act inference.
- Dataset is scripted, and digital states are LLM-synthesized, limiting ecological validity.
Experimental setup
Datasets: WAGIBench (scripted Aria recordings) Ego4D (related work comparison)
Benchmarks: WAGIBench
Metrics: MCQ accuracy LLM judge relevance score pairwise comparison accuracy with human raters
Baselines: Llama-3.2-11B-Vision Qwen2.5-VL-3B Qwen2.5-VL-7B Qwen2.5-VL-72B InternVL2.5-MPO-2B InternVL2.5-MPO-8B InternVL2.5-MPO-78B GPT-4.1 SBERT Similarity Snap-MCQ
Builds on: Ego4D Socratic models sentenceBERT DeepSeek-R1-Distill
Closest papers by meaning
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.87)
- MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models (ICML 2026 · similarity 0.86)
- WearVQA: A Visual Question Answering Benchmark for Wearables in Egocentric Authentic Real-world scenarios (NeurIPS 2025 · similarity 0.86)
- LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Tasks (CVPR 2026 · similarity 0.86)
- Mordal: Automated Pretrained Model Selection for Vision Language Models (ICLR 2026 · similarity 0.86)
- What matters when building vision-language models? (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.