Research map / Video / Video-language understanding
Pivot Correlational Neural Network for Multimodal Video Categorization
Sunghun Kang, Junyeong Kim, Hyunsoo Choi, Sungjin Kim, Chang D. Yoo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GRU with contextual GRU (cGRU) for the pivot hidden state (ablation) | Pivot CorrNN baseline · FCVID (C3D + VGGish) | mAP (%): 66.86 -> 67.57 | Improved |
| add pivot correlation maximization on hidden state (ablation) | Pivot CorrNN with cGRU · FCVID (C3D + VGGish) | mAP (%): 67.57 -> 67.68 | Improved |
| add pivot correlation maximization on prediction (ablation) | Pivot CorrNN with cGRU · FCVID (C3D + VGGish) | mAP (%): 67.57 -> 68.02 | Improved |
| maximize correlation on both hidden state and prediction (ablation) | Pivot CorrNN with cGRU · FCVID (C3D + VGGish) | mAP (%): 68.02 -> 68.45 | Improved |
| add adaptive aggregation module (ablation) | Pivot CorrNN with cGRU and pivot correlation · FCVID (C3D + VGGish) | mAP (%): 68.45 -> 69.54 | Improved |
| add all three proposed modules to the GRU baseline (ablation) | GRU baseline · FCVID (C3D + VGGish) | mAP (%): 66.86 -> 69.54 | Improved |
| add appearance features (CNN, SIFT) (ablation) | Pivot CorrNN · FCVID test | mAP (%): 69.54 -> 75.33 | Improved |
| add motion features IDT-HOF, IDT-HOG (ablation) | Pivot CorrNN · FCVID test | mAP (%): 75.33 -> 76.58 | Improved |
| add motion features IDT-MBH, IDT-Traj (ablation) | Pivot CorrNN · FCVID test | mAP (%): 76.58 -> 77.23 | Improved |
| add MFCC audio features (ablation) | Pivot CorrNN · FCVID test | mAP (%): 77.23 -> 77.60 | Improved |
| use Pivot CorrNN instead of best prior method rDNN | FCVID classifier · FCVID test | mAP (%): 76.0 -> 77.6 | Improved |
| use Pivot CorrNN without cGRU instead of logistic regression on concatenated features | video-level features · YouTube-8M (video-level features) | GAP (%): 76.79 -> 77.40 | Improved |
Closest papers by meaning
- Deep Multimodal Representation Learning From Temporal Data (CVPR 2017 · similarity 0.84)
- Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks (TPAMI 2017 · similarity 0.83)
- PIVOT: Prompting for Video Continual Learning (CVPR 2023 · similarity 0.82)
- PNeRV: Enhancing Spatial Consistency via Pyramidal Neural Representation for Videos (CVPR 2024 · similarity 0.82)
- Multimodal Graph Neural Architecture Search under Distribution Shifts (AAAI 2024 · similarity 0.81)
- Local-Global Multi-Modal Distillation for Weakly-Supervised Temporal Video Grounding (AAAI 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
