Research map / Large language models / Instruction and fine-tuning
Gradient Transformer: Learning to Generate Updates for LLMs
Binh-Nguyen Nguyen, Khang Tran, Hai Phan, Issa Khalil
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate LLM update vectors from TinyLM updates with Grad-Transformer instead of fine-tuned TinyLM alone | Qwen2.5-3B to 7B · AQuA-RAT | accuracy: 48.43 -> 61.02 | Improved |
| Use Grad-Transformer instead of Conf baseline | Qwen2.5-3B to 7B · Six datasets | average PGR: 58.94 -> 91.88 | Improved |
| Use Grad-Transformer instead of Conf baseline | Qwen2.5-3B to 7B · GSM8K | accuracy: 74.30 -> 73.59 | Hurt |
| Use Grad-Transformer instead of Conf baseline | Qwen2.5-3B to 7B · CommonsenseQA | accuracy: 83.46 -> 83.21 | Hurt |
| Aggregate five clients' TinyLM updates | Grad-Transformer · Multi-client PGR | PGR: SAMSum: Conf 98.24 vs ours 78.53; AQuA-RAT 34.70 -> 85.15 | Mixed |
| Apply DP-SGD with epsilon 0.1 to client TinyLM training | Grad-Transformer · DROP | accuracy: TinyLM 55.59 -> 44.70; Grad-Transformer 62.35 | Mixed |
| Shrink TinyLM from 3B to 0.5B (ablation) | Grad-Transformer to 7B LLM · AQuA-RAT | accuracy: 63.57 -> 63.28 | No clear effect |
| Scale LLM from 7B to 14B (ablation) | Grad-Transformer · AQuA-RAT | accuracy: 63.28 -> 67.63 (0.5B TinyLM) | Improved |
| Reverse the block order of update vectors (ablation) | Grad-Transformer · AQuA-RAT | accuracy: 63.57 -> 39.73 | Hurt |
| Train separate Grad-Transformers for query and value weights (ablation) | Grad-Transformer · AQuA-RAT | accuracy: 63.57 -> 63.61 | No clear effect |
| Increase samples per shadow dataset from 256 to 1024 (ablation) | Grad-Transformer · AQuA-RAT | accuracy: 57.87 -> 61.02 | Improved |
| Increase number of update vector tuples to 1,000 (ablation) | Grad-Transformer · AQuA-RAT | accuracy | Improved |
Closest papers by meaning
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.88)
- Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives (NeurIPS 2024 · similarity 0.88)
- PrivAuditor: Benchmarking Data Protection Vulnerabilities in LLM Adaptation Techniques (NeurIPS 2024 · similarity 0.88)
- Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix (NeurIPS 2025 · similarity 0.87)
- Bohdi: Heterogeneous LLM Fusion with Automatic Data Exploration (NeurIPS 2025 · similarity 0.87)
- Privacy-Preserving Instructions for Aligning Large Language Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
