Research map / Architectures and efficiency / Pruning and efficient tuning
ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends self-supervised pre-training of natural-image ViTs (DinoV2/MAE) to new domains like satellite imagery by unfreezing 1-2 blocks and LoRA-tuning the rest, then fine-tuning with LoRA or linear probing.
Problem
Pre-training vision foundation models from scratch on each new visual domain (e.g., satellite or medical images) is computationally expensive, and natural-image VFMs underperform under large domain shifts; PEFT typically only handles supervised fine-tuning, not continued unsupervised pre-training.
Approach
Continue the original SSL objective (DinoV2 or MAE) on the target domain with a parameter-efficient mix: fully unfreeze 1-2 ViT blocks (chosen by layer analysis) and apply LoRA to Q,V matrices of the remaining blocks, then fine-tune with LoRA or linear probing on labeled target data.
Key results
- On fMoW-RGB, D-[L]-r64 with LoRA-r8 fine-tuning reaches 79.28% top-1, a new SoTA over fully pre-trained models, using ~6% of ViT encoder parameters and 100 GPU hours versus 960+ for full pre-training.
- Linear probing on fMoW-RGB reaches 76.86% (77.48% with last-4 concatenation), an improvement of over 8% over prior SoTA linear probing.
- On fMoW-Sentinel, M-[1,L]-r32 reaches 60.15% top-1, competitive with fully pre-trained SatMAE (61.48%) at 290-320 vs 1150 pre-training GPU hours.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Extend DinoV2 pre-training on fMoW-RGB using ExPLoRA before LoRA-r8 fine-tuning | DinoV2 ViT-L · fMoW-RGB validation | top-1 accuracy: 78.08 -> 79.28 | Helped |
| Use ExPLoRA initialization for linear probing with last-4-block features | DinoV2 ViT-L · fMoW-RGB validation | linear-probing top-1 accuracy: 69.00 -> 77.48 | Helped |
| Extend MAE pre-training with ExPLoRA for multi-spectral satellite images | MAE ViT-L · fMoW-Sentinel validation | top-1 accuracy: 51.61 -> 60.15 | Helped |
| Use ExPLoRA initialization with LoRA fine-tuning | DinoV2 on iWildcam · iWildcam validation | top-1 accuracy: 67.10 -> 68.07 | Helped |
| Use ExPLoRA initialization with SSRT | DinoV2 ViT-B · VisDA-2017 validation | mean accuracy: 89.9 -> 90.9 | Helped |
| Use ExPLoRA initialization with linear probing | DinoV2 on iWildcam · iWildcam validation | top-1 accuracy: 66.04 -> 62.95 | Hurt |
| Use ExPLoRA initialization for SpaceNet-v1 segmentation | DinoV2 ViT-L · SpaceNet-v1 validation | mIoU: 76.69 -> 76.69 | No clear effect |
| Increase encoder LoRA rank from 8 to 32 (ablation) | DinoV2 ExPLoRA on fMoW-RGB · fMoW-RGB validation, linear probing | top-1 accuracy: +0.09% | No clear effect |
| Increase encoder LoRA rank from 32 to 64 (ablation) | DinoV2 ExPLoRA on fMoW-RGB · fMoW-RGB validation, linear probing | top-1 accuracy: +1.08% | Helped |
| Unfreeze blocks L-1 and L instead of only L (ablation) | DinoV2 ExPLoRA · fMoW-RGB validation, linear probing | top-1 accuracy: 77.48 -> 77.76 | Helped |
| Also unfreeze block 1 (ablation) | DinoV2 ExPLoRA with blocks L-1 and L unfrozen · fMoW-RGB validation, linear probing | top-1 accuracy: 77.76 -> 78.04 | Helped |
| Unfreeze blocks L-1 and L without LoRA (ablation) | DinoV2 extended pre-training · fMoW-RGB validation, linear probing | top-1 accuracy: 75.97 vs 76.31 | Hurt |
Surprising findings
- Full from-scratch DinoV2 pre-training on fMoW with 1200 GPU hours reached only 54.29% linear-probe accuracy, far below the natural-image initialization.
- On iWildcam, ExPLoRA linear probing underperforms DinoV2 (62.95 vs 66.04), attributed to a small domain gap since ImageNet already contains animals.
Limitations
- Gains on SpaceNet-v1 are limited, and the authors note extra pre-training is unclear given blacked-out regions and small dataset size.
- Linear probing underperforms DinoV2 on iWildcam, suggesting the method may overfit when the domain gap is small.
- Unfreezing block choice relies on layer analysis heuristics and was validated mainly on DinoV2 ViT-L; the authors suggest further study of other PEFT methods for pre-training and extension to LLMs.
Experimental setup
Datasets: fMoW-RGB fMoW-Sentinel fMoW-Temporal SpaceNet-v1 RESISC-45 Camelyon17 iWildcam GlobalWheat NAIP EuroSAT VisDA-2017 ImageNet
Benchmarks: fMoW-RGB fMoW-Sentinel fMoW-Temporal SpaceNet-v1 RESISC-45 Camelyon17 iWildcam GlobalWheat NAIP EuroSAT VisDA-2017
Metrics: top-1 accuracy mIoU AP@0.5:0.95 AR@0.5:0.95 GPU hours trainable parameters
Baselines: SatMAE ScaleMAE GFM GDA AdaLoRA BOFT VPT GVPT SA2VP Adapter+ Mona CrossScaleMAE
Builds on: LoRA DinoV2 MAE SatMAE iBOT
Closest papers by meaning
- Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation (NeurIPS 2024 · similarity 0.89)
- Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning (AAAI 2024 · similarity 0.89)
- PELA: Learning Parameter-Efficient Models with Low-Rank Approximation (CVPR 2024 · similarity 0.89)
- C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models (NeurIPS 2026 · similarity 0.88)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.88)
- Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.