Research map / Video / Action recognition
Context-Enhanced Memory-Refined Transformer for Online Action Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CMeRT, a transformer for online action detection and anticipation that adds near-past context in the encoder and generates near-future context from long-term memory in the decoder to fix a training-inference discrepancy in short-term memory.
Problem
Memory-based online action detection (OAD) methods train on all short-term frames with causal masks, creating imbalanced context and non-causal leakage via anticipation, which mismatches the inference setting that relies on the latest frame.
Approach
A context-enhanced encoder supplies near-past frames to early short-term frames, and a memory-refined decoder generates near-future context from compressed long-term memory (not short-term memory) to refine short-term representations without non-causal leakage.
Key results
- CMeRT reaches 73.2% mAP on THUMOS'14, versus 71.6% for MAT and 72.6% for JOAAD.
- CMeRT reaches 35.9% mAP on CrossTask, up from 33.9% for MAT.
- On EK100 CMeRT achieves 47.1% verb, 48.3% noun and 27.6% action Top-5 recall for detection, outperforming MAT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add near-future generation and memory refinement (ablation) | CMeRT · THUMOS’14 | mAP: 71.5 -> 73.0 | Helped |
| add near-future generation and memory refinement (ablation) | CMeRT · CrossTask | mAP: 33.4 -> 34.8 | Helped |
| add near-future generation and memory refinement (ablation) | CMeRT · EK100 | mean Top-5 Recall (action): 26.3 -> 27.1 | Helped |
| add context enhancement using near-past context (ablation) | CMeRT with memory refinement · THUMOS’14 | mAP: 73.0 -> 73.2 | Helped |
| add context enhancement using near-past context (ablation) | CMeRT with memory refinement · CrossTask | mAP: 34.8 -> 35.9 | Helped |
| add context enhancement using near-past context (ablation) | CMeRT with memory refinement · EK100 | mean Top-5 Recall (action): 27.1 -> 27.6 | Helped |
| vary near-past context length (ablation) | CMeRT · THUMOS’14 | mAP: 73.2; 72.8; 72.7; 72.4 | Mixed |
| vary near-past context length (ablation) | CMeRT · CrossTask | mAP: 35.1; 35.9; 35.6; 35.5 | Mixed |
| vary near-past context length (ablation) | CMeRT · EK100 | mean Top-5 Recall (action): 27.2; 27.3; 27.6; 27.5 | Mixed |
| vary near-future context length (ablation) | CMeRT · THUMOS’14 | mAP: 72.6; 73.0; 73.2; 72.7 | Mixed |
| vary near-future context length (ablation) | CMeRT · CrossTask | mAP: 35.1; 35.3; 35.9; 35.6 | Mixed |
| vary near-future context length (ablation) | CMeRT · EK100 | mean Top-5 Recall (action): 27.0; 27.6; 27.3; 27.1 | Mixed |
Surprising findings
- Naive MAT-stream training, which uses only the latest frame per short-term memory, drops performance sharply (e.g. 58.1% mAP on THUMOS'14).
- Distant (non-adjacent) past and future context consistently underperforms near context in ablations.
Limitations
- Struggles with short actions and small subjects in similar backgrounds, per qualitative analysis.
- Larger model and higher GFLOPS than LSTR and TeSTra, with lower FPS.
Experimental setup
Datasets: THUMOS'14 EPIC-Kitchens-100 CrossTask
Benchmarks: THUMOS'14 CrossTask EPIC-Kitchens-100
Metrics: mAP mean Top-5 Recall point-wise F1 segment-wise F1 edit score
Baselines: LSTR GateHub Testra MAT JOAAD MAT-rw MAT-stream IIU RULSTM TempAgg AVT
Builds on: LSTR MAT TeSTra GateHub
Closest papers by meaning
- Does Video-Text Pretraining Help Open-Vocabulary Online Action Detection? (NeurIPS 2024 · similarity 0.87)
- End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames (CVPR 2024 · similarity 0.86)
- Memory Consolidation Enables Long-Context Video Understanding (ICML 2024 · similarity 0.85)
- Dual DETRs for Multi-Label Temporal Action Detection (CVPR 2024 · similarity 0.85)
- OnlineTAS: An Online Baseline for Temporal Action Segmentation (NeurIPS 2024 · similarity 0.85)
- StreamForest: Efficient Online Video Understanding with Persistent Event Memory (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.