Research map / Video / Video-language understanding
Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TransFusion, a multimodal fusion transformer that anticipates next-active object interactions in egocentric video from a language summary of past action context plus the last frame, reporting SOTA on Ego4D short-term object interaction anticipation.
Problem
Short-term object interaction anticipation in egocentric video requires modeling the spatio-temporal action context formed by past actions on objects, which existing methods extract implicitly from fixed-length video chunks.
Approach
Summarize past action context textually as verb-noun pairs (from an image captioner) and CLIP-detected salient objects, aggregate them over frames, and fuse these language tokens with visual feature maps via joint self-attention in a Faster R-CNN-based detector.
Key results
- On Ego4D v2 test, TransFusion reaches 30.43 Noun mAP vs 26.15 for FRCNN+SF and 25.06 for StillFast.
- On Ego4D v1 test, TransFusion achieves 10.06 Noun-Verb mAP, above InternVideo (9.18) and StillFast (9.95).
- TransFusion outperforms TransFusion-Video (video features) across context lengths; overall N-V performance gain of about 21% reported over video features.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TransFusion instead of StillFast | Ego4D v1 object interaction anticipation · Ego4D v1 test | Overall mAP: 3.49 -> 3.54 | Helped |
| Use TransFusion instead of GANOv2 | Ego4D v2 object interaction anticipation · Ego4D v2 test | Noun mAP: 25.67 -> 30.43 | Helped |
| Combine action segments and salient objects as context (ablation) | TransFusion · Ego4D v1 validation | Noun-Verb mAP: 6.14 -> 7.55 | Helped |
| Remove language action context (ablation) | TransFusion · Ego4D v1 validation | Noun-Verb mAP: 7.55 -> 6.14 | Hurt |
| Use action segments as context (ablation) | TransFusion · Ego4D v1 validation | Noun-Verb mAP: 6.14 -> 7.16 | Helped |
| Replace generated context with ground-truth action labels (ablation) | TransFusion · Ego4D v1 validation | Noun-Verb mAP: 7.55 -> 7.86 | Helped |
| Replace SBERT with Flan-T5 language encoder (ablation) | TransFusion · Ego4D v1 validation | Noun-Verb mAP: 7.55 -> 6.61 | Hurt |
| Replace SBERT with GPT-2 language encoder (ablation) | TransFusion · Ego4D v1 validation | Noun-Verb mAP: 7.55 -> 7.08 | Hurt |
| Use TransFusion instead of TransFusion-Video (ablation) | Ego4D v1 validation · Ego4D v1 validation | Noun-Verb mAP: null | Helped |
Surprising findings
- Language summaries of past actions outperform video features (SlowFast) as action context representation for this task.
- SBERT (BERT/RoBERTa backbones) works better as language encoder than GPT-2 or Flan-T5.
Limitations
- Relies on off-the-shelf captioner and CLIP outputs that may be noisy, with domain gap between generated and ground-truth labels.
- Held-object representation performed worse than salient objects, and the approach does not match top action-anticipation methods on EK100 (out of scope).
Experimental setup
Datasets: Ego4D short-term object interaction anticipation v1 Ego4D short-term object interaction anticipation v2
Benchmarks: Ego4D short-term object interaction anticipation v1 Ego4D short-term object interaction anticipation v2
Metrics: Top-5 mAP Noun Top-5 mAP Noun-Verb Top-5 mAP Noun-TTC Top-5 mAP Noun-Verb-TTC (overall)
Baselines: FRCNN+SF InternVideo StillFast GANOv2 FRCNN+Rnd TransFusion-Video
Builds on: Faster R-CNN Sentence-BERT CLIP StillFast InternVideo
Closest papers by meaning
- Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions? (ICLR 2025 · similarity 0.89)
- Action Scene Graphs for Long-Form Understanding of Egocentric Videos (CVPR 2024 · similarity 0.87)
- FIction: 4D Future Interaction Prediction from Video (CVPR 2025 · similarity 0.87)
- Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos (CVPR 2025 · similarity 0.86)
- Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs (NeurIPS 2026 · similarity 0.86)
- MMEgo: Towards Building Egocentric Multimodal LLMs for Video QA (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.