Research map / Label-efficient and robust learning / Self-supervised and contrastive

VESSA: Video-based objEct-centric Self-Supervised Adaptation for Visual Foundation Models

NeurIPS 2025 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Pretext-task visual pre-training · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VESSA, a self-distillation fine-tuning method that adapts DINO, DINOv2 and TIPS to new object domains using only unlabeled multi-view object-centric videos, with LoRA, staged head/backbone unfreezing and uncertainty-weighted loss.

Problem

Adapting vision foundation models to specialized domains with distribution shift and no labels; continued self-supervised fine-tuning is common for language models but has been ineffective for vision encoders, which can forget pretrained knowledge.

Approach

Self-distillation fine-tuning that samples temporally separated frame pairs from short object-centric videos as positive views, trains the projection head before unfreezing the backbone, applies LoRA to early layers and full updates to last layers, and weights the DINO loss by teacher entropy (UWSD).

Key results

• On CO3D with DINOv2, VESSA reaches 91.85% k-NN accuracy vs 89.64% for ExPLoRA + video and 87.86% for the pretrained model.
• On MVImageNet, VESSA improves DINO from 90.44% (pretrained) to 92.51%, while ExPLoRA + video drops to 87.74% with DINO.
• Adapted models suffer catastrophic forgetting: ImageNet kNN accuracy for DINO drops from 76.10% to about 15.5% after VESSA adaptation.

Measured effects reported in the paper

Surprising findings

• Naively continuing self-supervised training on static domain images often degrades or does not improve performance relative to the pretrained model.
• Adding motion-inspired image augmentations to simulate camera movement did not recover the gains of real video frames.

Limitations

• Severe catastrophic forgetting: adapted models lose general-purpose ImageNet performance.
• Requires multi-view object-centric videos, which are not common in many real-world datasets.
• Cross-dataset generalization drops by about 5 to 7 points when trained on MVImageNet and tested on CO3D.

Experimental setup

Datasets: MVImageNet CO3D ImageNet

Benchmarks: CO3D MVImageNet ImageNet

Metrics: top-1 accuracy (k-NN, k=1)

Baselines: Pretrained foundation model ExPLoRA ExPLoRA + video Static-baseline DINO from scratch

Builds on: DINO LoRA ExPLoRA DINOv2

Closest papers by meaning

• Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.87)
• LoRA Recycle: Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs (CVPR 2025 · similarity 0.86)
• Interpreting Object-level Foundation Models via Visual Precision Search (CVPR 2025 · similarity 0.86)
• Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models (NeurIPS 2025 · similarity 0.86)
• Time- Memory- and Parameter-Efficient Visual Adaptation (CVPR 2024 · similarity 0.86)
• ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts (ICML 2025 · similarity 0.86)