Research map / Architectures and efficiency / Pruning and efficient tuning
Curvature Tuning: Provable Training-free Model Steering From a Single Parameter
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Curvature Tuning, which steers pretrained networks by replacing ReLUs with a single-parameter (beta) CTU activation, either training-free (S-CT) or as a parameter-efficient finetuning method with trainable beta and c (T-CT).
Problem
Finetuning and PEFT methods adapt weights and features, lack interpretability, and rely on heuristic hyperparameters; the paper asks how to steer pretrained models principledly and efficiently, targeting generalization and robustness.
Approach
Modulate the curvature of a network's decision boundary by injecting one hyperparameter beta into the activation functions (a convex combination of reparameterized SiLU and Softplus, the CTU), which provably projects the ReLU network onto a space of smooth functions without changing weights; trainable per-neuron (beta, c) gives T-CT.
Key results
- T-CT boosts average transfer accuracy of ResNet-50/152 by 8.59%/8.34% over linear probing and 4.64%/1.70% over LoRA (rank 1) across 12 datasets, using 11%-59% of LoRA's trainable parameters.
- S-CT, a single hyperparameter with no backprop, improves linf robust accuracy of ResNet-50/152 by 1032.64%/1494.46% relative to the frozen baseline on ImageNet/RobustBench benchmarks, with average linf gains reported in the table.
- On Swin-T/S, T-CT improves over linear probing by 0.48%/1.94% but underperforms LoRA by 4.01%/4.76%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ReLUs with CTUs using fixed c=0.5 and a shared tuned β | S-CT with ResNet-18 · 12 downstream datasets (average) | accuracy (%): 73.96 -> 75.34 | Helped |
| Replace ReLUs with CTUs using fixed c=0.5 and a shared tuned β | S-CT with ResNet-50 · 12 downstream datasets (average) | accuracy (%): 76.24 -> 76.92 | Helped |
| Train per-neuron β and c in CTUs | T-CT with ResNet-18 · 12 downstream datasets (average) | accuracy (%): 73.64 -> 78.26 | Helped |
| Train per-neuron β and c in CTUs | T-CT with ResNet-50 · 12 downstream datasets (average) | accuracy (%): 78.68 -> 81.31 | Helped |
| Replace ReLUs with CTUs and tune shared β | S-CT with ResNet-18/50/152 · RobustBench ℓ∞ attacks, average | robust accuracy (%): 44.01%, 1032.64%, and 1494.46% relative improvements for ResNet-18, ResNet-50, and ResNet-152 | Helped |
| Train per-neuron β and c in CTUs | T-CT with ResNet-18/50/152 · CIFAR-10/100 ℓ∞ attacks | robust accuracy (%): 445.00%, 1115.00%, and 493.02% over linear probing; 133.57%, 384.85%, and 2450.00% over LoRA | Helped |
| Train CTU parameters in transformer feed-forward blocks | T-CT with Swin-T · 12 downstream datasets (average) | accuracy (%): 0.48% over linear probing; trails LoRA by 4.01% | Mixed |
| Train CTU parameters in transformer feed-forward blocks | T-CT with Swin-S · 12 downstream datasets (average) | accuracy (%): 1.94% over linear probing; trails LoRA by 4.76% | Mixed |
| Use c=0.5 rather than c=1 (SiLU) (ablation) | S-CT with ResNet-18 · 12 downstream datasets (average) | accuracy (%): 1.99% | Helped |
| Use c=0.5 rather than c=0 (Softplus) (ablation) | S-CT with ResNet-18 · 12 downstream datasets (average) | accuracy (%): 0.56% | Helped |
| Use c=0.5 rather than c=1 (SiLU) (ablation) | S-CT with ResNet-50 · 12 downstream datasets (average) | accuracy (%): 0.94% | Helped |
| Use c=0.5 rather than c=0 (Softplus) (ablation) | S-CT with ResNet-50 · 12 downstream datasets (average) | accuracy (%): 0.57% | Helped |
Surprising findings
- Learned beta and c values in T-CT show sharp U-shaped distributions concentrated near 0 and 1, rather than near their means.
- T-CT, despite being a finetuning method, improves robustness without any robustness objective (implicit bias).
Limitations
- Theoretical guarantees rely on ReLU networks; attention and GELU in transformers fall outside the max-affine spline framework, and T-CT underperforms LoRA on Swin models.
- Currently CT is applied only to feed-forward blocks in transformers, leaving attention layers unmodified.
- Optimal beta for S-CT requires grid search over validation accuracy.
Experimental setup
Datasets: ImageNet Arabic Characters Arabic Digits Beans CUB-200-2011 DTD FashionMNIST FGVC-Aircraft Flowers102 Food101 PathMNIST OCTMNIST
Benchmarks: 12 downstream image classification datasets RobustBench l2 and linf adversarial benchmarks common corruption benchmark (CIFAR-10/100, ImageNet)
Metrics: accuracy robust accuracy relative improvement number of trainable parameters
Baselines: Linear probing (frozen) LoRA (rank 1) SiLU (c=1) Softplus (c=0) Full finetuning
Builds on: LoRA max-affine spline operators SiLU Softplus
Closest papers by meaning
- Stepping on the Edge: Curvature Aware Learning Rate Tuners (NeurIPS 2024 · similarity 0.88)
- CAMEx: Curvature-aware Merging of Experts (ICLR 2025 · similarity 0.86)
- The Sharp Directions Are Against You: Curvature Analysis of Activation Steering (NeurIPS 2026 · similarity 0.86)
- Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws (ICML 2025 · similarity 0.85)
- Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules (NeurIPS 2025 · similarity 0.85)
- Global curvature for second-order optimization of neural networks (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.