Research map / Video / Action recognition
Taylor Videos for Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Taylor videos, a motion-centric video format computed by Taylor-expanding frame differences within temporal blocks, which serves as a competitive or complementary input to RGB and optical flow for action recognition and skeleton-based recognition.
Problem
Extracting motion from video for action recognition is hard because motion has no explicit form, spans multiple concepts (displacement, velocity, acceleration), and is corrupted by noise from unstable pixels; optical flow is costly and RGB is redundant.
Approach
Treat motion as an implicit function of a temporal block and approximate it via Taylor series over frame differences and higher-order differences; the three leading terms give displacement, velocity, and acceleration channels forming a 3-channel Taylor frame that can replace RGB input in pretrained networks.
Key results
- With Kinetics pretraining, I3D on HMDB-51 reaches 78.1% with Taylor video versus 74.3% for RGB and 77.3% for optical flow.
- Taylor-transformed skeletons improve ST-GCN on NTU-60 X-Sub from 81.5% to 85.4%.
- Taylor video with TSM on SSv2 reaches 65.1% versus 63.4% for RGB when trained from scratch on large-scale data as reported in the large-scale table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace RGB input with Taylor video | I3D with Kinetics pretraining · HMDB-51 | top-1 accuracy: 74.3 -> 78.1 | Helped |
| Replace optical flow input with Taylor video | I3D with Kinetics pretraining · CATER moving | mAP: +3.5% | Helped |
| Replace RGB input with Taylor video | TSM trained from scratch · K400 | top-1 accuracy: 76.3 -> 77.6 | Helped |
| Replace RGB input with Taylor video | TSM trained from scratch · SSv2 | top-1 accuracy: 63.4 -> 65.1 | Helped |
| Apply Taylor transformation to skeleton sequences | ST-GCN · NTU-60 X-Sub | accuracy: 81.5 -> 85.4 | Helped |
| Apply Taylor transformation to skeleton sequences | 3Mformer · NTU-120 X-Set | accuracy: 93.8 -> 94.7 | Helped |
| Combine displacement, velocity, and acceleration instead of using displacement only (ablation) | TSM · Something-Something v2 | accuracy: 61.9 -> 65.1 | Helped |
| Combine displacement, velocity, and acceleration instead of using velocity only (ablation) | TSM · Something-Something v2 | accuracy: 62.1 -> 65.1 | Helped |
| Combine displacement, velocity, and acceleration instead of using acceleration only (ablation) | TSM · Something-Something v2 | accuracy: 59.9 -> 65.1 | Helped |
| Use raw frame-differencing maps instead of Taylor video (ablation) | TSM · Something-Something v2 | accuracy: 65.1 -> 53.9 | Hurt |
| Train with Taylor video from scratch rather than fine-tuning a pretrained model | Taylor video models · HMDB-51 | accuracy | Hurt |
| Increase temporal-block length and include more Taylor terms (ablation) | Taylor video · HMDB-51 | accuracy | Mixed |
Surprising findings
- Frame differencing alone performs worse than RGB on SSv2 (53.9% vs 59.1%), showing that combining higher-order motion terms is needed.
Limitations
- Taylor videos encode little static texture, so they are unsuitable for tasks like captioning and anomaly detection.
- Training from scratch on Taylor videos performs much worse than fine-tuning from pretrained RGB models.
- Taylor videos need more RGB frames than the output length (19 for 16-frame output), and transformer architectures require adaptation.
Experimental setup
Datasets: HMDB-51 MPII Cooking Activities CATER Kinetics-400 Kinetics-600 Something-Something v2 NTU RGB+D 60 NTU RGB+D 120 Kinetics-Skeleton
Benchmarks: HMDB-51 MPII Cooking Activities CATER Kinetics-400 Kinetics-600 Something-Something v2 NTU-60 NTU-120 Kinetics-Skeleton
Metrics: top-1 accuracy mean average precision compression ratio time per frame
Baselines: RGB optical flow GrayST frame differencing TVL1 optical flow
Builds on: optical flow dynamic images TSN TSM I3D
Closest papers by meaning
- Stitch, Contrast, and Segment: Learning a Human Action Segmentation Model Using Trimmed Skeleton Videos (AAAI 2025 · similarity 0.85)
- Disentangled Concepts Speak Louder Than Words: Explainable Video Action Recognition (NeurIPS 2025 · similarity 0.85)
- LookWhen? Fast Video Recognition by Learning When, Where, and What to Compute (NeurIPS 2026 · similarity 0.84)
- Revealing Key Details to See Differences: A Novel Prototypical Perspective for Skeleton-based Action Recognition (CVPR 2025 · similarity 0.84)
- Temporal Alignment-Free Video Matching for Few-shot Action Recognition (CVPR 2025 · similarity 0.84)
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.