Research map / Architectures and efficiency / RNNs and normalization
TrAct: Making First-layer Pre-Activations Trainable
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TrAct, which trains first-layer vision weights so their pre-activation updates match gradient descent on the embeddings, via a closed-form corrected weight update that speeds up training 1.25x-4x at small overhead.
Problem
In vision models, first-layer weight updates are proportional to input pixel values, so low-contrast or very bright/dark images affect first-layer learning unequally, unlike embedding layers in language models.
Approach
Perform a gradient step on first-layer pre-activations, then solve a regularized least-squares problem for the weight update, giving a closed form dW = -eta * grad_z x^T (xx^T/b + lambda I)^-1, implemented as a custom first-layer gradient without changing forward values or architecture.
Key results
- On CIFAR-100, TrAct (lambda=0.1) improves average top-1 accuracy over 36 CNNs from 75.90% to 76.39% at 200 epochs.
- On ImageNet, ResNet-50 with TrAct trained for 60 epochs (75.68% top-1) matches the 90-epoch baseline (75.70%), a 1.5x speedup.
- On DeiT-III ViT-S (400ep), TrAct raises top-1 from 81.23% to 81.50% with 0.08-0.25% runtime overhead in pretraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply TrAct to the first layer | 36 CNN architectures · CIFAR-100, 200 epochs, average | Top-1 accuracy: 75.90% -> 76.39% | Helped |
| Apply TrAct to the first layer | 36 CNN architectures · CIFAR-100, 200 epochs, average | Top-5 accuracy: 93.19% -> 93.42% | Helped |
| Apply TrAct to the first layer | ResNet-50 · ImageNet validation, 60 versus 90 epochs | Top-1 accuracy: -0.02% | No clear effect |
| Apply TrAct to the first layer | ResNet-50 · ImageNet validation, 60 versus 90 epochs | Top-5 accuracy: +0.04% | Helped |
| Apply TrAct during pre-training and fine-tuning | ViT-S · CIFAR-10, CIFAR-100, Flowers-102, and Stanford Cars transfer | Accuracy: 98.94% -> 99.02%; 90.70% -> 90.85%; 94.39% -> 95.58%; 90.44% -> 91.07% | Helped |
| Normalize each input patch to mean 0 and standard deviation 1 instead of using TrAct (ablation) | ViT · CIFAR-10, up to 400 epochs | Accuracy | Helped |
| Use patch normalization or DualPatchNorm instead of TrAct (ablation) | ViT · CIFAR-10, 800 epochs | Accuracy | Hurt |
| Use DualPatchNorm instead of TrAct (ablation) | ViT · CIFAR-10, 200 epochs | Accuracy | No clear effect |
| Remove dataset standardization (ablation) | ViT · CIFAR-10, 200 epochs, input ranges [0,1] and [0,255] | Accuracy | Mixed |
| Train the first layer with SGD while training the rest of the model with Adam (ablation) | ViT with TrAct · CIFAR-10 | Accuracy | Helped |
| Use λ=0.05 rather than the other tested λ values (ablation) | TrAct on ResNet-18 · CIFAR-10, most tested training lengths | Test accuracy | Helped |
| Use smaller λ values rather than larger λ values (ablation) | TrAct · Short versus long training | Accuracy | Mixed |
Surprising findings
- TrAct performs better with unnormalized [0,255] inputs than [0,1] inputs, and is nearly agnostic to lambda in that setting.
- Patch-wise normalization and DualPatchNorm do not improve over the baseline when training for 800 epochs, whereas TrAct still does.
Limitations
- Requires actually training the first layer, so it does not apply to frozen first layers (finetuning gains only shown with first layers unfrozen).
- Overhead can be substantial for small models (9.7% for the CIFAR-10 ViT) due to non-fused operations.
- Only one hyperparameter lambda; the good range may shift for different input scales, kernel sizes, or channel counts.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1k PASCAL VOC2007 Flowers-102 Stanford Cars
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1k PASCAL VOC2007 Flowers-102 Stanford Cars
Metrics: top-1 accuracy top-5 accuracy mean average precision training runtime
Baselines: vanilla SGD training vanilla Adam training patch-wise normalization DualPatchNorm DeiT III
Builds on: DeiT III Dual PatchNorm ISAAC K-FAC
Closest papers by meaning
- Accelerating Augmentation Invariance Pretraining (NeurIPS 2024 · similarity 0.86)
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.85)
- L-WISE: Boosting Human Visual Category Learning Through Model-Based Image Selection and Enhancement (ICLR 2025 · similarity 0.85)
- Perception Encoder: The best visual embeddings are not at the output of the network (NeurIPS 2025 · similarity 0.85)
- Beyond Random Augmentations: Pretraining with Hard Views (ICLR 2025 · similarity 0.85)
- Explicitly Modeling Subcortical Vision with a Neuro-Inspired Front-End Improves CNN Robustness (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.