Research map / Video / Video-language understanding
A Backpack Full of Skills: Egocentric Video Understanding with Diverse Task Perspectives
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EgoPack, a unified temporal-graph architecture for egocentric video tasks that stores multi-task prototypes as a reusable "backpack" to aid learning of a novel task on Ego4D.
Problem
Egocentric video understanding models are usually task-specific, and multi-task learning suffers negative transfer and requires supervision for all tasks at training time, limiting extension to new tasks later.
Approach
Shared GNN-based temporal backbone with task-specific heads; after multi-task pretraining, per-task feature prototypes (indexed by verb/noun pairs from action recognition) are frozen and stored, and a novel task queries them via k-NN message passing, with each task casting a vote on the final logits.
Key results
- EgoPack achieves 0.721 verb edit distance on LTA test, state of the art among methods trained without Ego4D pretraining (vs EgoT2 0.722 verb ED... and HierVL 0.724).
- On the validation set, EgoPack reaches 25.10% verb and 31.10% noun top-1 on AR and 71.83% OSCC accuracy, outperforming Temporal Graph single-task (24.25/30.43/71.26).
- PNR localisation error reduced from 1.76 s (MLP) to 0.61 s with temporal graph; EgoPack matches at 0.61 s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add temporal graph modelling to the MLP baseline | Ego4D temporal models · Ego4D PNR | localisation error (s): 1.76 s to 0.61 s | Helped |
| train all tasks jointly with hard parameter sharing (ablation) | Temporal Graph · Ego4D AR and OSCC | accuracy: 24.25 -> 22.05; 71.26 -> 71.10 | Hurt |
| use task-prototype cross-task interaction | EgoPack · Ego4D AR | verb top-1 accuracy (%): 24.25 -> 25.10 | Helped |
| add temporal graph modelling to the MLP baseline (ablation) | Ego4D tasks · Ego4D tasks (aggregated) | average score (∆): 0.416 -> 0.433 (+4.22%) | Helped |
| replace prototype-based interaction with task translation (ablation) | EgoPack temporal architecture · Ego4D tasks (aggregated) | average score (∆): 0.416 -> 0.431 (+3.61%) | Helped |
| add a multi-task objective and cross-task interaction (ablation) | Temporal Graph · Ego4D tasks (aggregated) | average score (∆): 0.416 -> 0.430 (+3.50%) | Helped |
| fine-tune a new task-specific head after multi-task training without other heads' outputs (ablation) | Multi-Task Learning model · Ego4D tasks (aggregated) | average score (∆): 0.416 -> 0.437 (+5.02%) | Helped |
| use task prototypes and cross-task interaction instead of MTL+FT (ablation) | EgoPack · Ego4D tasks (aggregated) | average score (∆): 0.437 -> 0.441 (+6.10%) | Helped |
| increase the number of nearest-neighbour task prototypes from k=0 to k=4 (ablation) | EgoPack · Ego4D tasks (aggregated) | average score | Helped |
| use EgoPack rather than EgoT2 for LTA verb prediction | EgoPack · Ego4D LTA test | verb edit distance: 0.722 -> 0.721 | Helped |
| use EgoPack rather than EgoT2 (EgoVLP) | EgoPack · Ego4D OSCC test | accuracy (%): 75.0 -> 72.1 | Hurt |
Surprising findings
- Naive multi-task learning underperforms single-task temporal graph models due to negative transfer, notably on AR and OSCC.
Limitations
- Gains on the validation set are modest (aggregate ∆ of about 6% over MLP) and comparisons on the test set are hard due to differing backbones and supervision.
- On OSCC test, EgoPack lags methods pretrained on full Ego4D (e.g., EgoVLP, EgoT2 with EgoVLP).
Experimental setup
Datasets: Ego4D
Benchmarks: Ego4D AR Ego4D LTA Ego4D OSCC Ego4D PNR
Metrics: top-1 accuracy edit distance temporal localisation error
Baselines: Ego4D baselines EgoT2 MLP Multi-Task Learning Task Translation EgoVLP HierVL I-CVAE
Builds on: GraphSAGE EgoT2 Omnivore TRN
Closest papers by meaning
- Action Scene Graphs for Long-Form Understanding of Egocentric Videos (CVPR 2024 · similarity 0.86)
- EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs (NeurIPS 2025 · similarity 0.86)
- Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs (NeurIPS 2026 · similarity 0.86)
- Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models (NeurIPS 2026 · similarity 0.85)
- Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (CVPR 2024 · similarity 0.85)
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.