Research map / Video / Video segmentation and tracking
Don't Judge by the Look: Towards Motion Coherent Video Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that hue variance helps video recognition and proposes Motion Coherent Augmentation (MCA), combining the efficient RGB-channel-swap-based SwapMix with Variation Alignment loss, to improve motion-focused video models.
Problem
Video recognition models overfit on small datasets, and hue jittering, a common augmentation, is avoided in object recognition because it changes appearance and is slow to compute in HSV space; its effect on video understanding is unstudied.
Approach
Hue variance is beneficial for video because motion matters more than static appearance. SwapMix cheaply alters hue by randomly permuting RGB channels and linearly mixing with the original video, and Variation Alignment aligns predictions on original and appearance-varied pairs (KL-style loss) so the model learns appearance-invariant representations.
Key results
- TSM+MCA improves TSM on Something-Something V1 by 1.94% (TSM 45.63% to 47.57% top-1 with lambda_AV=1).
- TSM+MCA reaches 81.87% on UCF101 (+2.30) and 51.63% on HMDB51 (+3.00) over TSM.
- Applying VA to AugMix and RandAugment improves them by 0.52% and 1.32% respectively on Something-Something V1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MCA | TSM · UCF101 | Acc1. (%): 79.57 -> 81.87; +2.30 | Helped |
| Add MCA | TSM · HMDB51 | Acc1. (%): 48.63 -> 51.63; +3.00 | Helped |
| Add MCA | TSM · Something-Something V2 | Acc1. (%): 59.29 -> 60.71; +1.42 | Helped |
| Add MCA | TSM · Kinetics400 | Acc1. (%): 70.28 -> 71.08; +0.80 | Helped |
| Add MCA | MoCo with I3D encoder · UCF101 | Acc1. (%): 63.68 -> 67.99; +4.31 | Helped |
| Pretrain with MCA rather than the listed augmentation baseline | TSM · HMDB51 linear probing | Acc1. (%): 63.59 -> 65.95 | Helped |
| Set lambda_AV to 1.00 (ablation) | TSM+MCA · Something-Something V1 | Acc1. (%): 45.63 -> 47.57 | Helped |
| Increase lambda_AV from 1.00 to 2.00 or 3.00 (ablation) | TSM+MCA · Something-Something V1 | Acc1. (%): 47.57 -> 47.03; 47.57 -> 46.57 | Hurt |
| Combine Background Erasing with MCA | TSM · Something-Something V1 | Acc1. (%): 47.57 -> 48.05 | Helped |
| Add MCA | TSM · ActivityNet | mAP (%): 73.10 -> 74.75; +1.65 | Helped |
| Reduce MCA application probability to rho 0.25 (ablation) | TSM+MCA · Something-Something V1 | Acc1. (%): 45.63 -> 47.26 | Helped |
| Use Hue Jittering instead of SwapMix (ablation) | TSM · Something-Something V1 | Acc1. (%) | Hurt |
Surprising findings
- Hue Jittering, a standard augmentation in object recognition, hurts CIFAR10 accuracy but improves video recognition on Something-Something V1.
- Cutout and CutMix degrade video recognition performance, attributed to breaking motion patterns.
Limitations
- Increased GPU memory during training because a second batch with different appearances is loaded.
- Longer training time since training pairs must be constructed for the AV loss.
Experimental setup
Datasets: Something-Something V1 Something-Something V2 UCF101 HMDB51 Kinetics400 ActivityNet v1.3 CIFAR10
Benchmarks: Something-Something V1 Something-Something V2 UCF101 HMDB51 Kinetics400 ActivityNet
Metrics: Top-1 accuracy ECE mAP training time
Baselines: TSM Hue Jittering Cutout CutMix Mixup VideoMix Background Erasing AugMix RandAugment Uniformer-S SlowFast
Builds on: TSM Uniformer CS-KD AugMix RandAugment
Closest papers by meaning
- VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models (ICML 2025 · similarity 0.85)
- Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation (NeurIPS 2024 · similarity 0.85)
- Spectral Motion Alignment for Video Motion Transfer Using Diffusion Models (AAAI 2025 · similarity 0.84)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.84)
- Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision (NeurIPS 2025 · similarity 0.84)
- VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.