Research map / Video / Action recognition
TIM: A Time Interval Machine for Audio-Visual Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Time Interval Machine (TIM), a multimodal transformer encoder that takes a long untrimmed audio-visual feature sequence plus a modality-specific time-interval query, and recognises the action in that interval; extended to detection with multi-scale dense queries.
Problem
Recognising actions in long audio-visual videos where audio and visual events have different temporal extents and labels; prior methods use trimmed clips or assume known action boundaries for context.
Approach
Elevates time intervals to first-class queries: a Time Interval MLP encodes continuous (start, end) pairs, which are concatenated to features and to modality-specific [CLS] query tokens, letting a transformer attend to the queried interval and untrimmed context across both modalities, with a temporal distance auxiliary loss.
Key results
- On EPIC-KITCHENS-100 validation, TIM reaches 57.5% top-1 action accuracy (verb 77.1, noun 67.2), beating M&M Mix's 53.6% by 3.9%.
- On EPIC-SOUNDS, TIM audio-visual reaches 58.3% top-1 vs. DiffSED's 56.9%.
- On Perception Test action recognition, TIM (A+V) reaches 61.1% vs. MTCN's 51.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add audio-visual TIM instead of M&M | EPIC-KITCHENS-100 recognition · EPIC-KITCHENS-100 val | Action top-1 accuracy: +3.9% | Helped |
| Add visual modality to audio recognition | TIM · EPIC-SOUNDS | Top-1 accuracy: +2.6% | Helped |
| Add audio modality to visual recognition | TIM · Perception Test | Visual recognition accuracy: +5.0% | Helped |
| Add visual modality to sound recognition | TIM · Perception Test | Sound recognition accuracy: +1.3% | Helped |
| Use audio-visual input instead of audio-only | TIM · AVE | Top-1 accuracy: +13.7% | Helped |
| Use learned positional encoding instead of interval encoding (ablation) | TIM · EPIC-KITCHENS-100 | Action top-1 accuracy: 57.5 -> 29.6 | Hurt |
| Use learned positional encoding instead of interval encoding (ablation) | TIM · EPIC-SOUNDS | Audio actions top-1 accuracy: 58.3 -> 23.7 | Hurt |
| Use sinusoidal positional encoding instead of interval encoding (ablation) | TIM · EPIC-SOUNDS | Audio actions top-1 accuracy: 58.3 -> 13.4 | Hurt |
| Encode only the interval centre instead of start and end (ablation) | TIM · EPIC-KITCHENS-100 | Action top-1 accuracy: 57.5 -> 55.6 | Hurt |
| Encode interval start and end separately, then add their encodings (ablation) | TIM · EPIC-KITCHENS-100 | Action top-1 accuracy: 57.5 -> 56.4 | Hurt |
| Encode start and end together, then add the encoding to the input (ablation) | TIM · EPIC-KITCHENS-100 | Action top-1 accuracy: 57.5 -> 56.9 | Hurt |
| Encode interval start and end separately, then concatenate their encodings (ablation) | TIM · EPIC-KITCHENS-100 | Action top-1 accuracy: 57.5 -> 57.1 | Hurt |
Surprising findings
- Sinusoidal and learned positional encodings perform far worse than the Time Interval MLP, with audio action accuracy of 13.4% and 23.7% versus 58.3% for the proposed encoding.
- Including the audio modality improves visual action accuracy on EPIC-KITCHENS-100 by only 0.9%, while audio-visual AVE gains 13.7% over audio-only.
Limitations
- Detection results are competitive rather than uniformly SOTA, trailing ActionFormer-style methods on some metrics such as sound localisation on Perception Test.
- Reported EPIC-KITCHENS gains are attributed to using a 224p input without LLMs or larger pre-training, so comparisons depend on backbone choice.
- Relies on pre-extracted frozen features rather than end-to-end video encoding.
Experimental setup
Datasets: EPIC-KITCHENS-100 EPIC-SOUNDS AVE Perception Test
Benchmarks: EPIC-KITCHENS-100 EPIC-SOUNDS AVE Perception Test
Metrics: top-1 accuracy average precision at IoU 0.1-0.5
Baselines: M&M Mix MTV LaViLa AVION MTCN MBT TBN ActionFormer G-TAD SSAST DiffSED PSP
Builds on: ActionFormer MTCN Omnivore VideoMAE Auditory SlowFast
Closest papers by meaning
- Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds (CVPR 2025 · similarity 0.87)
- Dense Audio-Visual Event Localization Under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration (AAAI 2025 · similarity 0.87)
- Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.87)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.87)
- LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos (CVPR 2025 · similarity 0.86)
- Dual DETRs for Multi-Label Temporal Action Detection (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.