Research map / Architectures and efficiency / Pruning and efficient tuning
SaMA: Morpho Adaptation via Asymmetric Expansion of Kronecker Product
An Nguyen, Anh Tong
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SaMA, a PEFT method that uses an asymmetric expansion of Kronecker products to overcome LoRA's flat-direction optimization bottleneck and linear rank scaling with parameter budget.
Problem
LoRA's invariance manifold creates flat directions in the loss landscape that hinder adaptive optimizers, and its effective rank grows only linearly with the parameter budget.
Approach
Replaces LoRA's low-rank update with an asymmetric expansion based on Kronecker products, which removes the invariance manifold responsible for flat directions and lets effective rank grow faster than linearly in the parameter budget.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase LoRA rank from 32 to 128 (ablation) | LoRA · DROP (Llama2-7B) | F1: 62.40 -> 58.82 | Hurt |
| use SaMA instead of LoRA at similar parameter budget (0.062%) | Llama2-7B · DROP | F1: 62.39 -> 66.61 | Improved |
| scale SaMA block size from 2,2 to 64,64 (ablation) | SaMA · DROP (Llama2-7B) | F1: 66.01 -> 68.07 | Improved |
| SaMA instead of QuanTA on Llama2-13B | Llama2-13B · DROP | F1: 69.0 -> 73.12 | Improved |
| use SaMA instead of ABBA | Mistral-7B · GSM8K | accuracy: 64.48 -> 67.70 | Improved |
| use SaMA 8,8 instead of ABBA on MATH | Gemma-2 9B · MATH | accuracy: 39.59 -> 39.36 | Hurt |
| use SaMA 4x4 instead of QuanTA with Q-V tuning | LLaMA-7B · Commonsense reasoning | average accuracy: 80.3 -> 81.9 | Improved |
| increase SaMA from 4x4 to 16x16 (ablation) | SaMA · Commonsense reasoning (LLaMA3-8B) | average accuracy: 87.4 -> 87.8 | Improved |
| increase SaMA from 4x4 to 16x16 (ablation) | SaMA · Commonsense reasoning (LLaMA-13B) | average accuracy: 85.5 -> 85.4 | No clear effect |
| expand local gauge group to dense 16-by-16 blocks (ablation) | SaMA · GSM8K and MATH | accuracy | Hurt |
| initialize L=0 and R random instead of R=0 and L random (ablation) | SaMA · DROP (Qwen2.5-1.5B) | F1: 39.41 -> 39.77 | Improved |
| use SaMA 2-2 instead of strongest PEFT baselines | ViT-B/16 · VTAB-1k | mean total accuracy: 74.5 -> 74.7 | Improved |
Experimental setup
Baselines: LoRA
Builds on: LoRA
Closest papers by meaning
- GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- RaSA: Rank-Sharing Low-Rank Adaptation (ICLR 2025 · similarity 0.87)
- Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape (ICML 2025 · similarity 0.87)
- RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models (NeurIPS 2025 · similarity 0.87)
- PoLAR: Polar-Decomposed Low-Rank Adapter Representation (NeurIPS 2025 · similarity 0.87)
- LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
