Research map / Video / Action recognition
Video Recognition in Portrait Mode
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PortraitMode-400, a 76k-video, 400-class dataset of portrait-mode short videos from Douyin, and empirically studies cross-format transfer, spatial bias, crop/augmentation recipes, temporal modeling, and audio for portrait video recognition.
Problem
Existing video recognition datasets are dominated by landscape-mode videos, so models and training recipes may be poorly suited to the portrait-mode short-form videos that are increasingly common on smartphones and social media.
Approach
A data-driven, hierarchical 400-category taxonomy built from Douyin search queries, with strictly single-label, human-verified annotation, plus controlled experiments comparing portrait and landscape subsets (S100-PM vs S100-LM from Kinetics-700) to expose format-specific spatial priors and recipe differences.
Key results
- On S100-PM, X3D-M trained on portrait-mode videos reaches 52.0% top-1 versus 44.5% when trained on landscape-mode videos.
- Random-scale Inception-style augmentation helps X3D-M (61.7% vs 61.2% on PM-400) but hurts transformers such as Uniformer-S on S100-PM (39.7% vs 42.0%).
- Adding temporal self-attention raises Uniformer-S accuracy by 4.6 points on PM-400 (45.7% to 50.3%), and adding audio raises PM-400 accuracy from 54.6% to 57.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use temporal self-attention instead of frame-wise logits with mean pooling | Uniformer-S · Kinetics-400 | top-1 accuracy: 72.1 -> 76.6; 4.5% | Helped |
| Use temporal self-attention instead of frame-wise logits with mean pooling | Uniformer-S · 3Massiv | top-1 accuracy: 41.9 -> 42.8; 0.9% | Helped |
| Use temporal self-attention instead of frame-wise logits with mean pooling | Uniformer-S · PM-400 | top-1 accuracy: 45.7 -> 50.3; 4.6% | Helped |
| Fuse audio with visual features | R3D-50 + VGG · 3Massiv | top-1 accuracy: 52.7 -> 54.9 | Helped |
| Fuse audio with visual features | R3D-50 + VGG · PM-400 | top-1 accuracy: 54.6 -> 57.0 | Helped |
| Use Inception-style resizing and area sampling instead of shorter-side resizing (ablation) | X3D-M · S100-PM | top-1 accuracy: 52.0 -> 54.2; 2.2% | Helped |
| Use Inception-style resizing and area sampling instead of shorter-side resizing (ablation) | X3D-M · 3Massiv | top-1 accuracy: 52.6 -> 53.7; 1.1% | Helped |
| Use Inception-style resizing and area sampling instead of shorter-side resizing (ablation) | X3D-M · PM-400 | top-1 accuracy: 61.2 -> 61.7; 0.5% | Helped |
| Use shorter-side resizing instead of Inception-style resizing and area sampling (ablation) | Uniformer-S · S100-PM | top-1 accuracy: 39.7 -> 42.0; 2.3% | Helped |
| Use shorter-side resizing instead of Inception-style resizing and area sampling (ablation) | Uniformer-S · 3Massiv | top-1 accuracy: 42.8 -> 43.6; 0.8% | Helped |
| Use shorter-side resizing instead of Inception-style resizing and area sampling (ablation) | Uniformer-S · PM-400 | top-1 accuracy: 50.2 -> 50.4 | Helped |
| Use shorter-side resizing instead of Inception-style resizing and area sampling (ablation) | MViTv2-S · S100-PM | top-1 accuracy: 36.9 -> 41.0 | Helped |
Surprising findings
- Inception-style random-scale augmentation degrades transformer models on portrait videos, contrary to landscape conventions.
- Audio alone scores only 15.2% on PM-400, yet its fusion with visual features still adds about 2.4 points.
Limitations
- S100-PM and S100-LM are drawn from Kinetics-700 rather than matched recordings, so content differs between formats.
- Attempts to convert landscape videos to portrait with AutoFlip gave poor quality, so a controlled same-content comparison was not achieved.
- Most experiments train from scratch without pretraining, which may not reflect pretrained-model practice.
Experimental setup
Datasets: PortraitMode-400 Kinetics-700 Kinetics-400 3Massiv ImageNet
Benchmarks: PortraitMode-400 S100-PM S100-LM 3Massiv Kinetics-400
Metrics: top-1 accuracy
Baselines: X3D-M Uniformer-S MViTv2-S 3Massiv
Builds on: Kinetics-700 Uniformer MViTv2 X3D
Closest papers by meaning
- Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation (ICLR 2025 · similarity 0.82)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.81)
- MMEgo: Towards Building Egocentric Multimodal LLMs for Video QA (ICLR 2025 · similarity 0.81)
- Dynamic Camera Poses and Where to Find Them (CVPR 2025 · similarity 0.81)
- Towards Understanding Camera Motions in Any Video (NeurIPS 2025 · similarity 0.81)
- MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos (CVPR 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.