Research map / Federated learning and privacy / Privacy attacks and secure learning
Split Adaptation for Pre-trained Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Split Adaptation (SA), which splits a pre-trained ViT into a low-bit quantized frontend sent to clients and a backend kept on the server, adding bi-level noise and OOD augmentations to adapt ViTs in few-shot settings while protecting data and model.
Problem
Adapting pre-trained ViTs to downstream tasks in privacy-sensitive settings where client data cannot be shared and the server model must not be exposed, under few-shot conditions.
Approach
Combines split learning with low-bit quantization of the frontend (calibrated on out-of-distribution Hilbert-transform augmented server data), client-side model and representation noise, and patch-retrieval augmentation of client representations, with backend quantization-aware tuning on the server.
Key results
- In 5-shot adaptation on CIFAR-100, SA reaches 81.98% vs 79.63% for Split Learning and 78.79% for Linear Probing.
- On DomainNet-Clipart 10-shot, SA achieves 71.13% vs 70.67% for Fine Tuning and 69.12% for Linear Probing.
- Against the FORA reconstruction attack on CIFAR-100, SA lowers SSIM to 0.23 from 0.80 for Split Learning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Split Adaptation instead of the strongest listed baseline | ViT-Large · CIFAR-100 3-shot, 5-shot, and 10-shot | accuracy: 74.05±1.21 -> 76.24±0.29; 79.63±0.53 -> 81.98±0.49; 83.66±0.27 -> 85.45±0.40 | Helped |
| use Split Adaptation instead of the strongest listed baseline | ViT-Large · Places365 3-shot, 5-shot, and 10-shot | accuracy: 26.84±1.03 -> 30.92±0.89; 34.25±1.45 -> 35.31±0.01; 35.78±0.30 -> 39.26±0.10 | Helped |
| use Split Adaptation instead of the strongest listed baseline | ViT-Large · DomainNet-Cl 3-shot, 5-shot, and 10-shot | accuracy: 58.14±0.63 -> 56.26±0.79; 63.95±1.18 -> 65.03±0.59; 70.67±0.72 -> 71.13±0.76 | Mixed |
| add Hilbert Transform data augmentation (ablation) | Split Adaptation · CIFAR-100, Places365, and DomainNet-Cl 5-shot | accuracy: 80.37 -> 81.98; 35.27 -> 35.31; 64.59 -> 65.03 | Helped |
| add out-of-distribution quantization-aware tuning (ablation) | Split Adaptation · CIFAR-100, Places365, and DomainNet-Cl 5-shot | accuracy: 79.03 -> 81.98; 29.50 -> 35.31; 64.42 -> 71.13 | Helped |
| add OOD quantization-aware tuning with simulated OOD frontends instead of using only the full-set quantized frontend (ablation) | Split Adaptation · CIFAR-100, Places365, and DomainNet-Cl 5-shot | accuracy: 80.05 -> 81.98; 29.59 -> 35.31; 64.16 -> 71.13 | Helped |
| add patch retrieval augmentation (ablation) | Split Adaptation · CIFAR-100, Places365, and DomainNet-Cl 5-shot | accuracy: 79.90 -> 81.98; 29.56 -> 35.31; 52.85 -> 71.13 | Helped |
| add representation-level Laplace noise (ablation) | Split Adaptation · CIFAR-100 data reconstruction defense against FORA | LPIPS: L:0.49 -> L:0.70 | Helped |
| add model-level Gaussian noise (ablation) | Split Adaptation · CIFAR-100 data reconstruction defense against FORA | LPIPS: L:0.67 -> L:0.70 | Helped |
Surprising findings
- The quantized frontend alone performs poorly (e.g., 26.34% on CIFAR-100), so the frontend's own utility is low and the backend tuning carries the performance.
- Model-level noise and representation-level Laplace noise act synergistically, with combined defense exceeding the sum of individual effects.
Limitations
- Relies on a small client-side representation extraction cost but the server still performs backend tuning and quantization calibration on server data.
- Evaluated only on image classification with ViT-Large, DeiT-Base and Swin-L; the privacy evaluation uses a single attack (FORA).
Experimental setup
Datasets: ImageNet CIFAR-100 Places365 DomainNet
Benchmarks: CIFAR-100 Places365 DomainNet-Clipart
Metrics: classification accuracy SSIM PSNR LPIPS
Baselines: Linear Probing Fine Tuning LN-TUNE Split Learning Offsite Tuning
Builds on: split learning offsite tuning uniform symmetric quantization FORA
Closest papers by meaning
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.88)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.87)
- ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts (ICML 2025 · similarity 0.87)
- Patch Rebirth: Fast and Transferable Model Inversion of Vision Transformers (NeurIPS 2026 · similarity 0.86)
- Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation (NeurIPS 2024 · similarity 0.86)
- Time- Memory- and Parameter-Efficient Visual Adaptation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.