Research map / Architectures and efficiency / Pruning and efficient tuning
Tangent Transformers for Composition,Privacy and Removal
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Tangent Attention Fine-Tuning (TAFT), which linearizes pre-trained ViTs via a single modified forward pass and fine-tunes only the tangent weights, enabling linear weight composition, unlearning, and DP benefits.
Problem
Fine-tuned Transformers are non-linear in weight space, so they lack convex training objectives, exact weight-space composition, and cheap unlearning; prior linearization worked only for small ResNets.
Approach
Derives closed-form JVP expressions for attention, LayerNorm, FC and GeLU so the first-order Taylor linearization of a pre-trained ViT can be computed in one modified forward pass, then fine-tunes only the linear tangent weights with a rescaled square loss and ridge penalty.
Key results
- TAFT-1 averages 84.3% across seven datasets versus 85.0% for non-linear fine-tuning of the last attention block (NLFT-1).
- With 50 shards composed by weight averaging, TAFT reaches 75.7% average accuracy versus 62.2% for Model Soup.
- For DP fine-tuning at epsilon=1 on CUB-200, linearized full-block fine-tuning reaches 46.3% versus 11.0% for non-linear fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune with TAFT instead of nonlinear fine-tuning across downstream visual classification tasks | ViT-L/16 · Caltech-256, MIT-67, Oxford Pets, Stanford Dogs, CUB-200, FGVC-Aircrafts, Stanford Cars | accuracy: TAFT-1: 95.8 -> 95.9; 88.1 -> 89.3; 94.2 -> 94.5; 91.2 -> 91.9; 88.8 -> 89.0; 69.9 -> 62.2; 67.3 -> 67.2 | Mixed |
| Compose shard models by combining TAFT weights rather than using Model Soup | TAFT composition · 7-dataset average, 10 shards | accuracy: 72.2 -> 81.3 | Helped |
| Compose shard models by combining TAFT weights rather than using Model Soup | TAFT composition · 7-dataset average, 25 shards | accuracy: 65.5 -> 78.5 | Helped |
| Compose shard models by combining TAFT weights rather than using Model Soup | TAFT composition · 7-dataset average, 50 shards | accuracy: 62.2 -> 75.7 | Helped |
| Use Rescaled Square Loss instead of CE or MSE (ablation) | TAFT · 7 downstream datasets | accuracy: +1.5% over CE; +9.0% over MSE | Helped |
| Reset the final attention block before linearization (ablation) | TAFT · 7 downstream datasets | accuracy: +1.5% | Helped |
| Keep the original pretrained attention weights instead of resetting the final attention block (ablation) | TAFT · Caltech-256 and MIT-67 | accuracy | Helped |
| Reset and linearize the CLS token (ablation) | TAFT · FGVC-Aircrafts | accuracy: +0.8% | Helped |
| Reset and linearize the CLS token (ablation) | TAFT · Stanford Cars | accuracy: +3.1% | Helped |
| Use projected gradient descent hard constraints instead of weight decay regularization (ablation) | TAFT · Stanford Cars, CUB, Aircrafts | accuracy | Hurt |
| Use CLIP followed by supervised ImageNet pretraining instead of ImageNet classification pretraining (ablation) | TAFT · MIT-67, Oxford Pets, Stanford Dogs | accuracy: 89.3 -> 91.8; 94.5 -> 95.1; 91.9 -> 95.2 | Helped |
| Use supervised classification pretraining instead of MAE pretraining (ablation) | TAFT · MIT-67, Oxford Pets, Stanford Dogs | accuracy: 75.4 -> 89.3; 90.0 -> 94.5; 74.7 -> 91.9 | Helped |
Surprising findings
- Linearized fine-tuning slightly outperforms non-linear fine-tuning on several datasets close to ImageNet, which the authors attribute to implicit regularization.
- Linearizing a randomly reset last attention block improves results on distant tasks, suggesting negative transfer from later pre-trained features.
Limitations
- Tangent models are strictly less expressive than the original non-linear model and degrade on tasks far from the pre-training data (e.g., Stanford Cars, FGVC-Aircrafts).
- Inference cost can be up to double that of the original model because of the modified forward pass.
- Performance depends on the linearization point, which must be chosen empirically; sharding hurts minority classes on imbalanced data.
Experimental setup
Datasets: ImageNet Caltech-256 MIT-67 Oxford Pets Stanford Dogs CUB-200 FGVC-Aircrafts Stanford Cars DTD
Benchmarks: Caltech-256 MIT-67 Oxford Pets Stanford Dogs CUB-200 FGVC-Aircrafts Stanford Cars DTD
Metrics: classification accuracy
Baselines: Non-linear fine-tuning (NLFT) Model Soup SISA Linearized ResNet-50 Adapter LoRA BitFit FC head fine-tuning
Builds on: Vision Transformer Rescaled Square Loss DP-SGD Model Soup SISA
Closest papers by meaning
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.85)
- Tuning Stable Rank Shrinkage: Aiming at the Overlooked Structural Risk in Fine-tuning (CVPR 2024 · similarity 0.85)
- Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach (CVPR 2024 · similarity 0.85)
- Transformers without Normalization (CVPR 2025 · similarity 0.85)
- Attention-Only Transformers via Unrolled Subspace Denoising (ICML 2025 · similarity 0.85)
- WeGeFT: Weight‑Generative Fine‑Tuning for Multi‑Faceted Efficient Adaptation of Large Models (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.