Research map / Video / Action recognition
Not All Layers Are Equal in Image-to-Video Transfer
Thinesh Thiyakesan Ponbagavathi, Constantin Seibold, Alina Roitberg
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows via layer-wise sensitivity analysis that temporal modeling needs differ across transformer layers in frozen image models adapted to video, motivating non-uniform adapter placement for parameter-efficient image-to-video transfer.
Problem
Parameter-efficient image-to-video transfer typically inserts identical temporal adapters at every transformer layer, assuming all layers need equal temporal modeling; this assumption is questioned.
Approach
A layer-wise sensitivity analysis across four video-native model families and two pretraining paradigms indicates that temporal sensitivity varies substantially across layers, so uniform adapter insertion is suboptimal.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace uniform adapter placement with sensitivity-guided spatial-then-temporal placement | ST-Adapter on DINOv2-B · IKEA-ASM | top-1 accuracy: 70.5 -> 77.6 | Improved |
| replace uniform placement with STRAP | M2-CLIP on DINOv2-B · HRI-30 | top-1 accuracy: 80.5 -> 92.3 | Improved |
| replace uniform placement with STRAP | Frame2Freq on CLIP-B/16 · Diving48 and IKEA-ASM | top-1 accuracy: 92.2 -> 91.1 (Diving48); 78.0 -> 76.5 (IKEA-ASM) | Hurt |
| replace uniform placement with STRAP | ST-Adapter, Frame2Freq, M2-CLIP · Fine-grained action benchmarks | trainable parameters: 7.3 -> 4.2 (ST-Adapter, DINOv2) | Improved |
| replace uniform placement with STRAP | M2-CLIP on DINOv2-B · Kinetics-400 and SSv2 | top-1 accuracy: 81.8 -> 81.9 (K400); 71.2 -> 71.6 (SSv2) | No clear effect |
| replace uniform placement with STRAP | adapters on DINOv2-L and CLIP-L · IKEA-ASM | top-1 accuracy: 80.1 -> 80.5 (DINOv2-L) | Improved |
| use the VideoMAE-derived boundary 6 on a MAE-B backbone instead of uniform placement | ST-Adapter on MAE-B · IKEA-ASM | top-1 accuracy: 43.2 -> 57.7 | Improved |
| reverse the order, temporal adapters early and spatial late (ablation) | STRAP placement · Drive&Act, IKEA-ASM, HRI-30 | top-1 accuracy: -11.1% (Drive&Act), -40.7% (IKEA-ASM), -15.7% (HRI-30) | Hurt |
| use width-adjusted uniform baseline with same parameter count as STRAP (ablation) | uniform placement · Drive&Act, IKEA-ASM, HRI-30 | top-1 accuracy: -13.3 (Drive&Act), -3.7 (IKEA-ASM), -3.5 (HRI-30) vs STRAP | Hurt |
| put both spatial and temporal adapters in every layer (12S/12T) (ablation) | STRAP placement · Frame2Freq | top-1 accuracy | Hurt |
| move boundary from 6S/6T to 10S/2T (ablation) | ST-Adapter on DINOv2 · HRI-30 | top-1 accuracy: 89.2 -> 80.6 | Hurt |
| use all-temporal uniform placement instead of 6S/6T (ablation) | ST-Adapter on DINOv2 · Drive&Act | top-1 accuracy: 81.9 -> 75.2 | Hurt |
Closest papers by meaning
- Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation? (NeurIPS 2026 · similarity 0.87)
- TTVidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining (NeurIPS 2026 · similarity 0.87)
- Efficient Transfer Learning for Video-language Foundation Models (CVPR 2025 · similarity 0.87)
- Time- Memory- and Parameter-Efficient Visual Adaptation (CVPR 2024 · similarity 0.86)
- A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames (CVPR 2024 · similarity 0.86)
- MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
