Research map / Video / Action recognition
Modality-Collaborative Test-Time Adaptation for Action Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MC-TTA, a multimodal (RGB + optical flow) video test-time adaptation method that uses teacher/student memory banks, source-friendly clip selection, multimodal prototype alignment and cross-modal relative consistency to adapt without source data.
Problem
Adapting multimodal video action recognition models to unlabeled target videos online when source data is unavailable, given multimodal and temporal domain shifts.
Approach
Builds teacher and student memory banks: the teacher stores source-friendly clip features selected by cross-modal prediction agreement and low entropy (SSFR) to form pseudo-source prototypes, while the student's target prototypes are aligned to them, plus a cross-modal relative consistency loss on inter-class prototype distance matrices across RGB and flow.
Key results
- On UCF-HMDBfull, MC-TTA reaches 88.61% (U→H) and 91.07% (H→U), improving over the best prior TTA baseline ViTTA by 2.22 and 1.93 points.
- On Epic-Kitchens, MC-TTA achieves a mean accuracy of 45.19% across six domain adaptation tasks, outperforming ViTTA (43.96%) by 1.23 points.
- Ablation on four datasets shows SSFR, multimodal prototype alignment and cross-modal relative consistency each add gains, with all three combined giving the best results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MC-TTA instead of ViTTA | I3D · UCF–HMDBfull U→H | accuracy: 86.39 -> 88.61 | Helped |
| Use MC-TTA instead of ViTTA | I3D · Epic-Kitchens D2→D1 | accuracy: 45.81 -> 45.46 | Hurt |
| Add SSFR (ablation) | MC-TTA without SSFR, Lma, or Lcmr · Epic-Kitchens | mean accuracy: 42.36 -> 42.65 | Helped |
| Add multimodal prototype alignment (Lma) (ablation) | MC-TTA with SSFR · Epic-Kitchens | mean accuracy: 42.65 -> 43.96 | Helped |
| Add cross-modal relative consistency (Lcmr) (ablation) | MC-TTA with SSFR · Epic-Kitchens | mean accuracy: 42.65 -> 44.52 | Helped |
| Add cross-modal relative consistency (Lcmr) (ablation) | MC-TTA with SSFR and Lma · Epic-Kitchens | mean accuracy: 43.96 -> 45.19 | Helped |
| Add SSFR (ablation) | MC-TTA without SSFR, Lma, or Lcmr · UCF–Olympic U→O | accuracy: 92.73 -> 92.73 | No clear effect |
| Use adaptive SSFR thresholds instead of manually selected thresholds (ablation) | MC-TTA · UCF–HMDBfull U→H | accuracy: 88.61 -> 88.06 | Hurt |
| Use adaptive SSFR thresholds instead of manually selected thresholds (ablation) | MC-TTA · Epic-Kitchens | mean accuracy: 45.19 -> 44.72 | Hurt |
| Set the SSFR modality-distance threshold α to 0.2–0.4 (ablation) | MC-TTA · UCF–HMDBfull | accuracy | Mixed |
| Set the SSFR confidence threshold β to 0.5–0.7 (ablation) | MC-TTA · UCF–HMDBfull | accuracy | Mixed |
| Use K ∈ {3, 5} selected features for pseudo-prototypes (ablation) | MC-TTA · UCF–HMDBfull | accuracy | Mixed |
Surprising findings
- SSFR gave no improvement on the small UCF-Olympic dataset, which the authors attribute to performance saturation.
- An adaptive threshold selection based on batch means underperformed manually tuned thresholds (44.72 vs 45.19 mean on Epic-Kitchens).
Limitations
- Cross-modal relative consistency gives smaller gains on Epic-Kitchens than on UCF-HMDBfull.
- SSFR thresholds (alpha, beta) are manually tuned; the adaptive alternative performs worse.
- Evaluated only on RGB and optical flow modalities with I3D backbones and four datasets.
Experimental setup
Datasets: UCF101 HMDB51 Olympic UCF-HMDBfull Epic-Kitchens
Benchmarks: UCF-Olympic UCF-HMDBsmall UCF-HMDBfull Epic-Kitchens
Metrics: accuracy
Baselines: Source-only Tent LAME T3A TSD ViTTA MM-TTA
Builds on: T3A TSD Tent I3D MM-TTA
Closest papers by meaning
- Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization (ICLR 2025 · similarity 0.88)
- A Multimodal, Multi-Task Adapting Framework for Video Action Recognition (AAAI 2024 · similarity 0.87)
- Unsupervised Video Domain Adaptation with Masked Pre-Training and Collaborative Self-Training (CVPR 2024 · similarity 0.87)
- MV-Adapter: Multimodal Video Transfer Learning for Video Text Retrieval (CVPR 2024 · similarity 0.86)
- Efficient Transfer Learning for Video-language Foundation Models (CVPR 2025 · similarity 0.86)
- Learning Causal Domain-Invariant Temporal Dynamics for Few-Shot Action Recognition (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.