Research map / Architectures and efficiency / RNNs and normalization
Learning from Offline Foundation Features with Tensor Augmentations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LOFF-TA, which caches frozen foundation-model (DINOv2, OpenCLIP) features and trains a compact DeiT-S-based classifier with spatial and Gaussian-noise tensor augmentations applied to the cached features, achieving up to 37x faster training and 26x lower memory than fine-tuning.
Problem
Adapting large vision foundation models is resource-intensive, and standard image augmentations cannot be used when training on cached foundation features because storing features of augmented images is prohibitive.
Approach
Train a lightweight classifier on cached features of non-augmented images and apply spatial tensor augmentations (flips, resize, shear, translate, rotate) plus Gaussian noise directly to the cached token tensors, with optional pooling to enable high-resolution inputs at no extra training cost.
Key results
- LOFF-TA with pooling on ViT-G reaches 91.8% kappa on APTOS, 94.6% on AID, 96.3% AUC on DDSM, 79.9% recall on ISIC, and 90.1% on NABirds, beating unfrozen+linear on 6 of 10 settings.
- LOFF-TA trains up to 37x faster than fine-tuning the foundation model (218 vs 6 images/s on ViT-G) and uses up to 26x less GPU memory (13.2 GB vs 345.2 GB).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add tensor augmentations to cached features | LOFF · High-resolution APTOS2019 | κ: 89.6 $\pm$ 0.2 -> 90.4 $\pm$ 0.6 | Helped |
| Add tensor augmentations to cached features | LOFF · High-resolution AID | accuracy: 91.9 $\pm$ 0.3 -> 92.3 $\pm$ 0.7 | Helped |
| Use 512-resolution features with max pooling instead of 256-resolution features without pooling | LOFF-TA · High-resolution APTOS2019 | κ: 90.4 $\pm$ 0.6 -> 90.5 $\pm$ 1.0 | Helped |
| Add the foundation model CLS token (ablation) | LOFF-TA · Oxford-III Pet | accuracy: 94.1 -> 95.2 | Helped |
| Add the foundation model CLS token (ablation) | LOFF-TA · Caltech-101 | accuracy: 95.6 -> 96.4 | Helped |
| Add layer normalization (ablation) | LOFF-TA · Oxford-III Pet | accuracy: 94.9 -> 95.2 | Helped |
| Add layer normalization (ablation) | LOFF-TA · Caltech-101 | accuracy: 94.8 -> 96.4 | Helped |
| Add spatial tensor augmentations (ablation) | LOFF-TA · Oxford-III Pet | accuracy: 94.9 -> 95.1 | Helped |
| Add spatial tensor augmentations (ablation) | LOFF-TA · Caltech-101 | accuracy: 95.4 -> 96.2 | Helped |
| Add Gaussian noise to features with spatial augmentations (ablation) | LOFF-TA · Oxford-III Pet | accuracy: 95.1 -> 95.2 | Helped |
| Add Gaussian noise to features with spatial augmentations (ablation) | LOFF-TA · Caltech-101 | accuracy: 96.2 -> 96.4 | Helped |
| Replace spatial and noise tensor augmentations with Trivial Augment (ablation) | LOFF-TA · Oxford-III Pet | accuracy: 95.2 -> 95.4 | Helped |
Surprising findings
- Tensor augmentations such as vertical flips harm SUN397 accuracy much less than image vertical flips (-0.2% vs -2%).
- LOFF-TA with pooling outperforms full fine-tuning of the foundation model in 6 of 10 high-resolution settings.
Limitations
- Full fine-tuning generally outperforms LOFF-TA given sufficient compute.
- Inference is less efficient than a standalone foundation model because of the classifier on top.
- Mismatched augmentations like horizontal flips are not semantically exact in feature space; benefits are empirical.
Experimental setup
Datasets: APTOS2019 AID DDSM ISIC NABirds Flowers102 StanfordCars StanfordDogs Oxford-III Pet Caltech-101 SUN397 ImageNet
Benchmarks: APTOS2019 AID DDSM ISIC NABirds Flowers102 StanfordCars StanfordDogs Oxford-III Pet Caltech-101 SUN397
Metrics: accuracy quadratic weighted kappa AUC recall throughput (images/s) GPU memory (GB)
Baselines: Frozen + linear Unfrozen + linear Frozen + DeiT-S VPT AdaptFormer SSF LOFF
Builds on: DINOv2 DeiT AdamW VPT AdaptFormer SSF
Closest papers by meaning
- A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models (ICML 2025 · similarity 0.84)
- Low-Resource Vision Challenges for Foundation Models (CVPR 2024 · similarity 0.84)
- Accessing Vision Foundation Models via ImageNet-1K (ICLR 2025 · similarity 0.84)
- LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation (ICLR 2025 · similarity 0.83)
- Interfacing Foundation Models' Embeddings (NeurIPS 2024 · similarity 0.83)
- Foundation Models Secretly Understand Neural Network Weights: Enhancing Hypernetwork Architectures with Foundation Models (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.