AI Paper Atlas

Research map / Video / Action recognition

Not All Layers Are Equal in Image-to-Video Transfer

Thinesh Thiyakesan Ponbagavathi, Constantin Seibold, Alina Roitberg

NeurIPS 2026 · Video › Action recognition · approach: Contrastive video representation learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows via layer-wise sensitivity analysis that temporal modeling needs differ across transformer layers in frozen image models adapted to video, motivating non-uniform adapter placement for parameter-efficient image-to-video transfer.

Problem

Parameter-efficient image-to-video transfer typically inserts identical temporal adapters at every transformer layer, assuming all layers need equal temporal modeling; this assumption is questioned.

Approach

A layer-wise sensitivity analysis across four video-native model families and two pretraining paradigms indicates that temporal sensitivity varies substantially across layers, so uniform adapter insertion is suboptimal.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace uniform adapter placement with sensitivity-guided spatial-then-temporal placementST-Adapter on DINOv2-B · IKEA-ASMtop-1 accuracy: 70.5 -> 77.6Improved
replace uniform placement with STRAPM2-CLIP on DINOv2-B · HRI-30top-1 accuracy: 80.5 -> 92.3Improved
replace uniform placement with STRAPFrame2Freq on CLIP-B/16 · Diving48 and IKEA-ASMtop-1 accuracy: 92.2 -> 91.1 (Diving48); 78.0 -> 76.5 (IKEA-ASM)Hurt
replace uniform placement with STRAPST-Adapter, Frame2Freq, M2-CLIP · Fine-grained action benchmarkstrainable parameters: 7.3 -> 4.2 (ST-Adapter, DINOv2)Improved
replace uniform placement with STRAPM2-CLIP on DINOv2-B · Kinetics-400 and SSv2top-1 accuracy: 81.8 -> 81.9 (K400); 71.2 -> 71.6 (SSv2)No clear effect
replace uniform placement with STRAPadapters on DINOv2-L and CLIP-L · IKEA-ASMtop-1 accuracy: 80.1 -> 80.5 (DINOv2-L)Improved
use the VideoMAE-derived boundary 6 on a MAE-B backbone instead of uniform placementST-Adapter on MAE-B · IKEA-ASMtop-1 accuracy: 43.2 -> 57.7Improved
reverse the order, temporal adapters early and spatial late (ablation)STRAP placement · Drive&Act, IKEA-ASM, HRI-30top-1 accuracy: -11.1% (Drive&Act), -40.7% (IKEA-ASM), -15.7% (HRI-30)Hurt
use width-adjusted uniform baseline with same parameter count as STRAP (ablation)uniform placement · Drive&Act, IKEA-ASM, HRI-30top-1 accuracy: -13.3 (Drive&Act), -3.7 (IKEA-ASM), -3.5 (HRI-30) vs STRAPHurt
put both spatial and temporal adapters in every layer (12S/12T) (ablation)STRAP placement · Frame2Freqtop-1 accuracyHurt
move boundary from 6S/6T to 10S/2T (ablation)ST-Adapter on DINOv2 · HRI-30top-1 accuracy: 89.2 -> 80.6Hurt
use all-temporal uniform placement instead of 6S/6T (ablation)ST-Adapter on DINOv2 · Drive&Acttop-1 accuracy: 81.9 -> 75.2Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy