Research map / Architectures and efficiency / Pruning and efficient tuning
Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SketchTune, which compresses LLM weights into learned, fine-tunable sketches (shared quantized values plus a one-hot mapping) so compression and adaptation happen in one unified, adapter-free framework.
Problem
Parameter-efficient fine-tuning of LLMs relies on low-rank adapters that are restrictive and, when combined with quantized base weights, yield suboptimal quality and two unmergeable computation paths that slow training and inference.
Approach
Replace additive low-rank adapters with a learned sketching of each weight row (weighted k-means on Hessian-weighted parameters plus GPTQ-style error compensation), so the compressed weights themselves are the trainable parameters; a theorem characterizes when sketching beats low-rank approximation for weight updates.
Key results
- SketchTune GPR=8 (INT4) reaches 70.3 average on LLaMA-7B math vs LoRA 69.7 and Full FT 71.0 while using 3.4x smaller base models.
- On GSM8K at 2-bit, SketchTune GPR=4 reaches 39.88 vs LoftQ 25.40 for Llama-2-7B (with 7.3x fewer trainable parameters per the paper's reported comparison).
- On commonsense reasoning with Llama-3-8B, SketchTune GPR=4 averages 87.0 vs S2FT 86.6 and DoRA 85.2 while using a 5.92 GB base instead of 16.06 GB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SketchTune GPR=4 INT4 instead of LoftQ rank 64 NF4 | Llama-2-7B fine-tuning · GSM8K | accuracy: 35.00 -> 39.73 | Helped |
| Use SketchTune GPR=4 INT4 instead of LoftQ rank 64 NF4 | Llama-2-13B fine-tuning · GSM8K | accuracy: 45.00 -> 50.34 | Helped |
| Use SketchTune GPR=4 INT2 instead of LoftQ rank 64 NF2 | Llama-2-7B fine-tuning · GSM8K | accuracy: 20.90 -> 29.95 | Helped |
| Use SketchTune GPR=4 INT2 instead of LoftQ rank 64 NF2 | Llama-2-13B fine-tuning · GSM8K | accuracy: 25.40 -> 39.88 | Helped |
| Use SketchTune GPR=1 INT2 instead of LoftQ rank 64 NF2 | Llama-2-7B fine-tuning · WikiText-2 | perplexity: 7.85 -> 7.40 | Helped |
| Use SketchTune GPR=4 instead of S²FT | Mistral-7B instruction fine-tuning · MT-Bench | average score: 5.27 -> 6.36 | Helped |
| Increase GPR from 1 to 8 (ablation) | SketchTune on math reasoning · LLaMA-7B math reasoning average | accuracy: 68.9 -> 70.3 | Helped |
| Increase GPR from 1 to 8 (ablation) | SketchTune on math reasoning · LLaMA-13B math reasoning average | accuracy: 71.2 -> 73.4 | Helped |
| Increase GPR from 1 to 8 (ablation) | SketchTune on math reasoning · LLaMA2-7B math reasoning average | accuracy: 71.0 -> 73.1 | Helped |
| Increase GPR from 1 to 8 (ablation) | SketchTune on math reasoning · LLaMA3-8B math reasoning average | accuracy: 78.2 -> 79.9 | Helped |
| Increase GPR from 1 to 8 (ablation) | SketchTune on WikiText-2 · Llama-7B INT4 | perplexity: 5.82 -> 5.77 | Helped |
| Use INT2 instead of INT4 (ablation) | SketchTune GPR=4 · WikiText-2, Llama-7B | perplexity: 5.82 -> 12.37 | Hurt |
Surprising findings
- Sketched INT4 base models with only trainable sketch parameters match or beat full-precision LoRA/DoRA/S2FT on several benchmarks despite being 2.6-3.5x smaller.
- The weight updates of fine-tuned models are far from low-rank: an average rank above 1000 is needed to explain 75% of variance.
Limitations
- Sketching adds a one-time preprocessing step per base model (tens of minutes to hours).
- Results on math and commonsense are mostly on older Llama models; some benchmarks show SketchTune below baselines (e.g., AQuA and ARC-c).
Experimental setup
Datasets: Math10K Commonsense170K Alpaca-GPT4 WikiText-2 GSM8K C4
Benchmarks: MultiArith GSM8K AddSub AQuA SingleEq SVAMP MAWPS BoolQ PIQA SIQA HellaSwag WinoGrande
Metrics: accuracy perplexity MT-Bench score time to first token decoding latency GPU memory normalized approximation error
Baselines: LoRA DoRA S2FT Galore LoReFT LISA Full FT QLoRA LoftQ
Builds on: LoRA GPTQ LoftQ QLoRA S2FT
Closest papers by meaning
- Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.89)
- LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgrades (NeurIPS 2025 · similarity 0.89)
- Optimization Inspired Few-Shot Adaptation for Large Language Models (NeurIPS 2025 · similarity 0.88)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- Dynamic Low-Rank Sparse Adaptation for Large Language Models (ICLR 2025 · similarity 0.88)
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.