Research map / Video / Video-language understanding
Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic study of PEFT and probing for low-resource video tasks, comparing image- vs video-pretrained backbones and distributing temporal context across backbone, PEFT and probe.
Problem
How to adapt foundation models to video in low-data regimes, where PEFT and probing are attractive, and where temporal reasoning should be placed (backbone, PEFT modules, probe) remains unclear.
Approach
Treats temporal context as a resource that can be allocated separately to backbone, PEFT and probe (different frame counts per component), and benchmarks standard PEFT against image-to-video PEFT on appearance, motion and spatial low-resource tasks.
Key results
- LoRA achieves the best overall PEFT performance on appearance and motion tasks (e.g. 62.1 top-1 on CAER, 68.7 on IndustReal) while AdaptFormer leads spatial tasks (VSPW mIoU 50.1).
- Standard PEFT (AdaptFormer, LoRA) on the video-pretrained InternVideo-Next outperforms image-to-video ST-Adapter and image backbones on most tasks, with the largest gains on motion.
- Temporal context in the backbone drives motion performance (14 of 19% drop from reducing internal temporal modeling), while spatial tasks benefit from backbone temporal context but not from a temporal probe.
Surprising findings
- DoRA does not outperform LoRA on video despite being more recent.
- Spatial tasks degrade when the probe integrates over many frames, so temporal context should be minimized in the probe for dense prediction.
Limitations
- Temporal context is varied one or two components at a time; joint search over the configuration space is left open.
- Evaluation limited to six datasets in specific domains and mostly ViT-B backbones.
Experimental setup
Datasets: CAER NurViD IndustReal MammAlps ScanNet VSPW
Benchmarks: CAER NurViD IndustReal MammAlps ScanNet VSPW
Metrics: top-1 accuracy AbsRel mIoU throughput
Baselines: Full Finetuning Attentive Probe Linear Probe Efficient Probe ST-Adapter AdaptFormer LoRA DoRA VPT VFPT BPT BitFit
Builds on: LoRA AdaptFormer ST-Adapter VPT DoRA DiST DPT Video Depth Anything V-JEPA 2
Closest papers by meaning
- TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models (ICLR 2025 · similarity 0.88)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.88)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.88)
- SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding (CVPR 2025 · similarity 0.87)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.87)
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.