Research map / Label-efficient and robust learning / Recognition and long-tail
LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LEVI, a layer-wise ensemble that combines a frozen or light-tuned large pre-trained model's intermediate layers with a small trained-from-scratch model via adapting layers, to improve OOD generalization of fine-tuning.
Problem
Fine-tuned pre-trained models often generalize poorly OOD, because spurious or missing features can come from both the pre-trained model and the fine-tuning data, and pre-training data is unavailable during fine-tuning.
Approach
Layer-wise ensemble that connects intermediate outputs of a large pre-trained model with a small randomly initialized task-specific model through small adapting layers, each producing a prediction trained with an averaged loss, so the two complementary views suppress problematic features from both sources while keeping training and inference cheap.
Key results
- On MovieLens genre shift, LEVI reaches OOD RMSE 1.065 vs 1.268 for T5x full fine-tuning and 1.127 for the best baseline FT+RobustModel.
- On ImageNet-Sketch-trained ViT, LEVI achieves the best OOD accuracy of 33.62 vs 32.10 for FT+ZeroShot.
- LEVI with LoRA-tuned ViT improves OOD accuracy on Medical from 73.06 to 73.57 and on ImageNet from 22.08 to 25.64.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LEVI instead of FT+RobustModel | T5x recommendation · MovieLens OOD (12 genres) | RMSE: 1.127 -> 1.065 | Helped |
| Use LEVI instead of FT+FS | T5x recommendation · Amazon Review OOD (7 products) | RMSE: 1.521 -> 1.298 | Helped |
| Use LEVI instead of FT+ZeroShot | ViT image classification · ImageNet OOD | accuracy: 32.10 -> 33.62 | Helped |
| Add LEVI to a LoRA-tuned ViT | ViT image classification · ImageNet ID | accuracy: 79.71 -> 78.78 | Hurt |
| Add LEVI to a LoRA-tuned ViT | ViT image classification · ImageNet OOD | accuracy: 22.08 -> 25.64 | Helped |
| Replace the layer-wise ensemble with a final-prediction ensemble of two T5x models (ablation) | LEVI · MovieLens OOD (genre shifts) | RMSE: 1.065 -> 1.178 | Hurt |
| Replace the layer-wise ensemble with a final-prediction ensemble of T5x and MLP (ablation) | LEVI · MovieLens OOD (genre shifts) | RMSE: 1.065 -> 1.163 | Hurt |
| Remove the trained-from-scratch task-specialized model (ablation) | LEVI · MovieLens OOD (genre shifts) | RMSE: 1.065 -> 1.142 | Hurt |
| Remove large-model features and use a single-head MLP (ablation) | LEVI · MovieLens OOD (genre shifts) | RMSE: 1.065 -> 1.158 | Hurt |
| Remove large-model features and use a multi-head MLP (ablation) | LEVI · MovieLens OOD (genre shifts) | RMSE: 1.065 -> 1.121 | Hurt |
Surprising findings
- Full fine-tuned models can generalize OOD worse than both pre-trained-only and trained-from-scratch models, motivating the joint-view design.
- FT+ZeroShot sometimes largely fails to improve OOD performance on recommendation tasks, suggesting pre-trained language features may be insufficient there.
Limitations
- Evaluated only on T5x-small and ViT-base with a small set of OOD shifts; scaling to larger models is untested.
- Relies on the assumption that spurious features in pre-training and fine-tuning data do not overlap, as stated in the paper's remark.
- Ablation and sensitivity results are reported mainly for recommendation tasks.
Experimental setup
Datasets: MovieLens Amazon Review Diabetic Retinopathy ImageNet-Sketch
Benchmarks: MovieLens genre shift Amazon Review time and product shifts Diabetic Retinopathy quality shift ImageNet-A ImageNet-R ImageNet-V2
Metrics: RMSE accuracy
Baselines: T5x fine-tuning LP→FT FT+RobustModel FT+ZeroShot FT+FS LoRA-tuned ViT
Builds on: LP→FT robust fine-tuning LoRA
Closest papers by meaning
- Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods (ICML 2025 · similarity 0.89)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.88)
- Two-stage LLM Fine-tuning with Less Specialization and More Generalization (ICLR 2024 · similarity 0.88)
- PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.88)
- Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks (ICLR 2024 · similarity 0.88)
- Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.