Research map / Video / Action recognition
Action Dubber: Timing Audible Actions via Inflectional Flow
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Audible Action Temporal Localization and the Audible623 dataset, and proposes TA²Net, which uses an optical-flow-derived inflectional flow (second derivative of motion) plus self-supervised spatial contrastive/smoothness losses to locate sound-producing collision frames without audio.
Problem
Localizing frames where visible actions (e.g., collisions) would produce sound, to support automatic dubbing of silent videos, where existing action recognition and temporal action localization methods do not capture fine-grained kinematic changes at collision moments.
Approach
Treats the second temporal derivative of optical-flow motion (inflectional flow, a per-frame acceleration proxy) as a kinematic prior signaling collisions, fuses it with motion and appearance features via cross-kinematics attention, and adds a self-supervised spatial motion-localization objective (inter-video contrastive plus intra-video temporal smoothness) as auxiliary training.
Key results
- On Audible623, TA²Net reaches F1 0.616, precision 0.656, NME 3.462 and PME 0.744, outperforming the best baseline TransRAC (F1 0.553) on the four key metrics.
- Zero-shot on UCFRep and CountixAV counting, TA²Net achieves MAE 0.588 and 0.549, ranking first on CountixAV (OBO 0.346) without any counting training.
- Ablation: adding motion flow and inflectional flow raises F1 from 0.406 to 0.561 over the appearance-only baseline; full model reaches F1 0.616.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TA²Net instead of the strongest reported baseline | Audible action temporal localization · Audible623 | F1: 0.553 -> 0.616 | Helped |
| Use TA²Net instead of the strongest reported baseline | Audible action temporal localization · Audible623 | NME: 9.327 -> 3.462 | Helped |
| Use TA²Net instead of TransRAC | Repetitive action counting · UCFRep | MAE: 0.594 -> 0.588 | Helped |
| Use TA²Net instead of TransRAC | Repetitive action counting · UCFRep | OBO: 0.222 -> 0.185 | Hurt |
| Use TA²Net instead of TransRAC | Repetitive action counting · CountixAV | MAE: 0.686 -> 0.549 | Helped |
| Use TA²Net instead of TransRAC | Repetitive action counting · CountixAV | OBO: 0.255 -> 0.346 | Helped |
| Add motion flow to video-only features (ablation) | TA²Net · Audible623 | F1: 0.406 -> 0.501 | Helped |
| Add inflectional flow to video + motion features (ablation) | TA²Net · Audible623 | F1: 0.501 -> 0.561 | Helped |
| Add cross-kinematics aggregation (ablation) | TA²Net · Audible623 | F1: 0.561 -> 0.587 | Helped |
| Add inter-video contrastive loss (ablation) | TA²Net with aggregation · Audible623 | F1: 0.587 -> 0.601 | Helped |
| Add intra-video temporal regularization (ablation) | TA²Net with contrastive loss · Audible623 | F1: 0.601 -> 0.616 | Helped |
| Use only temporal regularization, without contrastive loss (ablation) | TA²Net with aggregation · Audible623 | Precision: 0.597 -> 0.627 | Helped |
Surprising findings
- Zero-shot transfer: a model trained only on Audible623 matches or beats RepNet and TransRAC, which were trained for repetitive counting, on CountixAV.
- STG-NF achieves the highest recall (0.679) but very low precision (0.229), and video-LLM Video-LLaMA fails to localize audible action timing.
Limitations
- Accuracy depends on optical flow quality; large viewpoint changes degrade the inflectional flow and cause missed detections.
- Assumes audible actions are visible collisions and does not use audio, so it cannot handle sounds without visible kinematic cause.
- Dataset is small (623 videos, 14 categories) and built from Kinetics, UCF101 and YouTube.
Experimental setup
Datasets: Audible623 Kinetics UCF101 UCFRep CountixAV AVE
Benchmarks: Audible623 UCFRep CountixAV AVE
Metrics: Recall Precision F1 NME PME MAE OBO
Baselines: BMN ActionFormer TriDet RepNet TransRAC TimeSformer X3D Hiera STG-NF
Builds on: GMFlow ResNet50 focal loss
Closest papers by meaning
- Weakly-Supervised Temporal Action Localization by Inferring Salient Snippet-Feature (AAAI 2024 · similarity 0.86)
- Bridge the Gap: From Weak to Full Supervision for Temporal Action Localization with PseudoFormer (CVPR 2025 · similarity 0.86)
- Towards Open-Vocabulary Audio-Visual Event Localization (CVPR 2025 · similarity 0.85)
- Dense Audio-Visual Event Localization Under Cross-Modal Consistency and Multi-Temporal Granularity Collaboration (AAAI 2025 · similarity 0.85)
- TIM: A Time Interval Machine for Audio-Visual Action Recognition (CVPR 2024 · similarity 0.84)
- Energy vs. Noise: Towards Robust Temporal Action Localization in Open-World (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.