Research map / Large language models / Instruction and fine-tuning
ScaleOT: Privacy-utility-scalable Offsite-tuning with Dynamic LayerReplace and Selective Rank Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ScaleOT, an offsite-tuning framework that uses RL-estimated layer importance to replace less important LLM layers with lightweight harmonizers, plus SVD-based rank compression of attention layers, to produce privacy-utility-scalable emulators.
Problem
Offsite-tuning shares a lossy compressed emulator of an LLM with data owners; existing methods (uniform LayerDrop, knowledge distillation) suffer adaptation degradation, high compute cost, and weak privacy protection.
Approach
Estimate per-layer importance with an RL-style update while training lightweight harmonizers that replace less important layers (Dynamic LayerReplace), then apply selective low-rank (SVD) compression to MHSA weights of the remaining frozen layers to strengthen privacy with little plug-in loss.
Key results
- On GPT2-XL, ScaleOT w/o SRC plug-in averages 50.1 vs. full fine-tuning 49.9 and OT 48.3 across eight QA tasks.
- With SRC, ScaleOT emulator zero-shot average drops to 30.1 (from 37.7 for OT and 41.9 for the raw model) on GPT2-XL, while plug-in stays at 49.3.
- On LLaMA-7B, ScaleOT w/o SRC plug-in averages 61.2 vs. OT 54.1 and CRaSh 57.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ScaleOT instead of OT | GPT2-XL · Eight QA benchmarks (Avg.) | accuracy: 48.3 -> 49.3 | Helped |
| Use ScaleOT instead of OT | OPT-1.3B · Eight QA benchmarks (Avg.) | accuracy: 49.0 -> 49.9 | Helped |
| Add SRC (ablation) | ScaleOT on GPT2-XL · Eight QA benchmarks (Avg.) | accuracy: 50.1 -> 49.3 | Hurt |
| Add SRC (ablation) | ScaleOT on OPT-1.3B · Eight QA benchmarks (Avg.) | accuracy: 50.3 -> 49.9 | Hurt |
| Add SRC (ablation) | ScaleOT on OPT-6.7B · Eight QA benchmarks (Avg.) | accuracy: 56.8 -> 55.5 | Hurt |
| Add SRC (ablation) | ScaleOT on LLaMA-7B · Eight QA benchmarks (Avg.) | accuracy: 61.2 -> 58.2 | Hurt |
| Use ScaleOT instead of CRaSh | OPT-6.7B · Eight QA benchmarks (Avg.) | accuracy: 55.3 -> 55.5 | Helped |
| Use ScaleOT instead of CRaSh | LLaMA-7B · Eight QA benchmarks (Avg.) | accuracy: 57.1 -> 58.2 | Helped |
| Add Adapter (d=64) (ablation) | ScaleOT w/o SRC on GPT2-XL · WikiText-2 | perplexity: 14.60 -> 14.92 | Hurt |
| Add LoRA (r=4) (ablation) | ScaleOT w/o SRC on GPT2-XL · WikiText-2 | perplexity: 14.60 -> 14.53 | Helped |
| Add Adapter (d=64) (ablation) | ScaleOT on GPT2-XL · WikiText-2 | perplexity: 15.95 -> 16.03 | Hurt |
| Add LoRA (r=4) (ablation) | ScaleOT on GPT2-XL · WikiText-2 | perplexity: 15.95 -> 15.85 | Helped |
Surprising findings
- Plug-in performance of ScaleOT can exceed direct full fine-tuning on the LLM, attributed to sparse training convergence.
- Rank compression of MHSA layers degrades emulator performance much more than plug-in performance, enabling a privacy gap with little utility loss.
Limitations
- Compression still requires training, limiting cost-effectiveness.
- Results on OPT-6.7B show ScaleOT with SRC underperforming its w/o SRC variant on several tasks.
- Privacy is assessed only via emulator performance gap, not formal privacy guarantees.
Experimental setup
Datasets: Pile WikiText OBQA PIQA ARC-E ARC-C HellaSwag SciQ WebQuestions RACE
Benchmarks: WikiText OBQA PIQA ARC-E ARC-C HellaSwag SciQ WebQuestions RACE
Metrics: perplexity accuracy
Baselines: OT CRaSh Full fine-tuning Zero-shot
Builds on: offsite-tuning (OT) LayerDrop CRaSh LISA LoRA
Closest papers by meaning
- DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer (ICLR 2024 · similarity 0.87)
- Scaling Laws for Differentially Private Language Models (ICML 2025 · similarity 0.86)
- Privacy-Preserving Instructions for Aligning Large Language Models (ICML 2024 · similarity 0.86)
- Split-and-Denoise: Protect large language model inference with local differential privacy (ICML 2024 · similarity 0.86)
- Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix (NeurIPS 2025 · similarity 0.85)
- DPZero: Private Fine-Tuning of Language Models without Backpropagation (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.