Research map / Architectures and efficiency / Pruning and efficient tuning
SAN: Hypothesizing Long-Term Synaptic Development and Neural Engram Mechanism in Scalable Model's Parameter-Efficient Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAN, a plug-and-play PEFT add-on that decomposes per-layer scaling vectors from adapters (LoRA/SSF/DoRA) and explicitly propagates them into the next layer's frozen weights, motivated by Neural Engram and LTP/D analogies.
Problem
Parameter-efficient fine-tuning (PEFT) methods with few trainable parameters often lack expressiveness relative to full fine-tuning, especially under larger domain gaps, and their shifts are approximately low-rank.
Approach
Compute a scaling vector from the ratio between LoRA-updated and original layer outputs, pool it, and explicitly multiply the subsequent layer's frozen weights by it (W'_{l+1} = gamma_l ⊙ W_{l+1}), giving finer-grained per-parameter adjustment and an implicit regularizer without extra trainable parameters.
Key results
- On FGVC with ImageNet-21k ViT-B, SAN+SSF reaches 91.6% mean accuracy vs. 90.7% for SSF and 88.5% for FFT, with no extra trainable parameters.
- On commonsense reasoning, SAN+LoRA-32 on LLaMA3-8B reaches 85.4% mean accuracy vs. 80.8% for LoRA-32 (+4.6), and SAN+DoRA reaches 85.5%.
- On visual instruction tuning with LLaVA1.5-7B, SAN+DoRA reaches 68.9% mean accuracy vs. 66.5% for FFT and 67.7% for DoRA-128.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SAN propagation to SSF | ViT-B vision fine-tuning · FGVC | Mean accuracy (%): 90.7 -> 91.6 (+0.9) | Helped |
| Add SAN propagation to SSF | ViT-B vision fine-tuning · VTAB-1k | Mean accuracy (%): 76.1 -> 77.7 (+1.6) | Helped |
| Add SAN propagation to SSF | ViT-B vision fine-tuning · GIC CIFAR100 | Accuracy (%): 94.0 -> 94.2 (+0.2) | Helped |
| Add SAN propagation to SSF | ViT-B vision fine-tuning · GIC ImageNet-1k | Accuracy (%): 83.1 -> 83.8 (+0.7) | Helped |
| Add SAN propagation to LoRA | LLaMA-7B · CSR-170k | Mean accuracy (%): 74.7 -> 78.0 (+3.2) | Helped |
| Add SAN propagation to DoRA | LLaMA-7B · CSR-170k | Mean accuracy (%): 78.0 -> 78.8 (+0.8) | Helped |
| Add SAN propagation to LoRA | LLaMA3-8B · CSR-170k | Mean accuracy (%): 80.8 -> 85.4 (+4.6) | Helped |
| Add SAN propagation to DoRA | LLaMA3-8B · CSR-170k | Mean accuracy (%): 85.2 -> 85.5 (+0.3) | Helped |
| Add SAN propagation to DoRA | LLaVA1.5-7B · MVIT-665K | Mean accuracy (%): 67.7 -> 68.9 (+1.2) | Helped |
| Add SAN propagation to LoRA | LLaVA1.5-7B · MVIT-665K | Mean accuracy (%): 67.0 -> 68.0 (+1.0) | Helped |
| Use randomly selected posterior layers instead of logically proximate layers (ablation) | SAN+LoRA · Vision and language tasks | Relative improvement | Mixed |
| Apply SAN to Q, UP, and Down layers only instead of Q, K, V, UP, and Down (ablation) | SAN+LoRA on LLaMA-7B · CSR-170k | Mean accuracy (%): 78.0 -> 75.8 | Hurt |
Surprising findings
- Random (non-topological) propagation of scaling vectors gives unstable results, sometimes hurting performance, suggesting propagation target choice matters.
- PEFT with SAN on LLaVA1.5-7B reportedly outperforms FFT on the larger LLaVA1.5-13B model (67.8% average).
Limitations
- Theoretical justification relies on linear-network approximations and near-linear activation assumptions that are not rigorously proven for transformers.
- Hyperparameters such as learning rate were searched manually or per-dataset, and the choice of propagation layers (Q,K,V,UP,Down) strongly affects results.
- Biological Neural Engram/LTP-D analogy is hypothesized rather than empirically validated in neuroscience terms.
Experimental setup
Datasets: CUB-Birds NA-Birds Oxford Flowers Stanford Dogs Stanford Cars VTAB-1k CIFAR-100 ImageNet-1k CSR-170k MVIT-665K RoboVQA
Benchmarks: FGVC VTAB-1k CIFAR-100 ImageNet-1k BoolQ PIQA SIQA HellaSwag WinoGrande ARC-E ARC-C OpenBookQA
Metrics: accuracy mean accuracy trainable parameter percentage
Baselines: Full Fine-Tuning Linear Probing ChatGPT (CoT) VPT NOAH SSF Bitfit LoRA DoRA SPT-LoRA Adapter AdaptFormer
Builds on: LoRA SSF DoRA Visual Prompt Tuning NOAH
Closest papers by meaning
- Learning the Plasticity: Plasticity-Driven Learning Framework in Spiking Neural Networks (NeurIPS 2025 · similarity 0.85)
- Crafting Reversible SFT Behaviors in Large Language Models (NeurIPS 2026 · similarity 0.85)
- HetSyn: Versatile Timescale Integration in Spiking Neural Networks via Heterogeneous Synapses (NeurIPS 2025 · similarity 0.85)
- Structured flexibility in recurrent neural networks via neuromodulation (NeurIPS 2024 · similarity 0.85)
- Model Based Inference of Synaptic Plasticity Rules (NeurIPS 2024 · similarity 0.84)
- Discovering plasticity rules that organize and maintain neural circuits (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.