Research map / Large language models / Instruction and fine-tuning

Cross-model Control: Improving Multiple Large Language Models in One-time Training

NeurIPS 2024 · Large language models › Instruction and fine-tuning · approach: Calibration and knowledge in LLM tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Cross-model Control (CMC), training a tiny delta language model alongside a frozen template LLM so its logit shifts can be reused across LLMs of different sizes and vocabularies, with a PM-MinED token mapping.

Problem

Fine-tuning outcomes are tied to a single model's parameters, so improving many LLMs with different sizes and vocabularies requires repeated costly training; the paper asks whether one portable model can alter logits of many LLMs.

Approach

Observes that logit shifts from fine-tuning are highly similar across models; trains a ~110M tiny delta model added to the log-softmax of a frozen template LLM, then reuses it on other LLMs via a prefix-match plus minimum-edit-distance token mapping (PM-MinED).

Key results

• On AlpacaEval, a single 110M delta model on Llama2-7b, Llama2-13b, Llama2-70b and Mistral-7B raises win rate by +26.19, +31.51, +38.26 and +26.46 points respectively (e.g., 30.41, 39.04, 49.81, 33.29), outperforming Proxy-tuning which cannot handle different vocabularies.
• On TOFU unlearning, CMC on Llama2-13b-TOFU reduces forget-set ROUGE-L to 0.00 and truth ratio to 0.29 while keeping retain-set ROUGE-L at 0.97, comparable to LoRA and better than delta-Unlearning at under half its inference cost.
• Smaller delta models (15M) retain much of the gain, e.g., 40 win% on Llama2-7b and 72 on Llama2-70b on the first 50 AlpacaEval instructions.

Measured effects reported in the paper

Surprising findings

• On Llama2-70b, a larger delta model (110M) gave lower win rate than a 42M one, which the authors attribute to overfitting.
• A 15M-parameter delta model still yields substantial gains, suggesting very small models suffice to steer large LLMs.

Limitations

• Delta model performance is constrained by its vocabulary; tokens from languages not covered by the delta model's vocabulary are not adjusted.
• Performance lags LoRA on instruction tuning, since LoRA is model-specific while the delta model is portable.
• No error bars due to the cost of GPT-4-based AlpacaEval evaluation.

Experimental setup

Datasets: GPT4-Alpaca Stanford Alpaca TOFU GSM8k

Benchmarks: AlpacaEval TOFU

Metrics: win rate ROUGE-L probability truth ratio Sinkhorn divergence

Baselines: LoRA fine-tuning Proxy-tuning delta-Unlearning Vanilla base model

Builds on: LoRA Proxy-tuning DExperts delta-Unlearning

Closest papers by meaning

• Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.90)
• Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.89)
• When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.89)
• Beyond Next Token Prediction: Patch-Level Training for Large Language Models (ICLR 2025 · similarity 0.89)
• Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch (ICML 2024 · similarity 0.89)
• In-Context Unlearning: Language Models as Few-Shot Unlearners (ICML 2024 · similarity 0.89)