Research map / Large language models / Instruction and fine-tuning

Transformer-Squared: Self-adaptive LLMs

ICLR 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Transformer-Squared, a self-adaptive LLM framework that trains RL-optimized singular-value scaling vectors (SVF) per task and mixes them at inference via prompt-based, classifier-based, or CEM few-shot dispatch.

Problem

Fine-tuning LLMs for many tasks is costly, static, prone to overfitting and task interference, and expert modules are hard to compose at runtime.

Approach

Singular Value Fine-tuning (SVF) learns one scaling vector per weight matrix over its singular values, trained with REINFORCE on correctness rewards; a two-pass inference scheme then dispatches or interpolates these expert vectors (via prompting, a trained classifier expert, or CEM-searched few-shot mixing) to adapt the base weights per prompt.

Key results

• On unseen tasks with Llama3-8B-Instruct, Transformer^2 (Few-shot) reaches 25.47 on MATH (vs 24.54 base, 24.12 LoRA), 62.99 on Humaneval (vs 60.98 base) and 82.61 on ARC-Challenge (vs 80.63 base).
• SVF on Llama3-8B-Instruct improves GSM8K to 79.23 (MLP+attention, 0.58M params) vs 75.89 base, while LoRA with RL collapses to 57.92.
• Mistral-7B-Instruct-v0.3 with Transformer^2 (Few-shot) reaches 47.40 on Humaneval vs 43.29 base, while LoRA drops to 37.80.

Measured effects reported in the paper

Surprising findings

• SVF expert vectors trained on Llama3-8B-Instruct transfer positively to Mistral-7B-Instruct-v0.3 when singular-value ordering is preserved, but shuffling them degrades performance.
• Standard LoRA trained with RL is highly unstable and collapses early in training across a wide learning-rate sweep.

Limitations

• Expert capabilities are bounded by the latent components of the base model.
• CEM-based few-shot adaptation adds one-time overhead that may be significant for tasks with few prompts.
• Cross-model transfer may depend on architectural similarity between the two models.

Experimental setup

Datasets: GSM8K MBPP-pro ARC-Easy TextVQA MATH Humaneval ARC-Challenge OKVQA

Benchmarks: GSM8K MBPP-pro ARC-Easy TextVQA MATH Humaneval ARC-Challenge OKVQA

Metrics: accuracy normalized score relative to base model

Baselines: LoRA IA3 DoRA LoRA-XS Llama3-8B-Instruct Mistral-7B-Instruct-v0.3 Llama3-70B-Instruct

Builds on: LoRA REINFORCE cross-entropy method IA3 DoRA

Closest papers by meaning

• Self-Adapting Language Models (NeurIPS 2025 · similarity 0.89)
• Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning (NeurIPS 2024 · similarity 0.87)
• Efficient Large Language Model Inference with Neural Block Linearization (NeurIPS 2025 · similarity 0.87)
• Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing (NeurIPS 2024 · similarity 0.87)
• On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention for Long-Context LLM Serving (ICML 2025 · similarity 0.87)
• LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.87)