Research map / Architectures and efficiency / Pruning and efficient tuning

Portable Reward Tuning: Towards Reusable Fine-Tuning across Different Pretrained Models

ICML 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Parameter-efficient transfer analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Portable Reward Tuning (PRT), which trains an explicit reward model with the standard cross-entropy fine-tuning loss so it can be reused with any newer pretrained model sharing the label/vocabulary space, avoiding the three-model inference overhead of emulated fine-tuning.

Problem

Fine-tuned models must be retrained whenever the underlying foundation model is replaced by a newer or larger one, and existing inference-time tuning (emulated fine-tuning) reuses a fine-tuned model but needs three models at inference.

Approach

Reparameterize fine-tuning as KL-regularized reward maximization: train an explicit reward network r_theta with the standard cross-entropy loss through the PRT model softmax(log pi_pt + r_theta), then at inference combine r_theta with any new pretrained model's log-probabilities, requiring only two models instead of three.

Key results

• PRT transferring a reward trained on Llama3.2-1B to Llama3-8B improves speed to 0.75x of pretrained vs 0.7x for EFT on time per token, with lower inference memory than EFT in vision experiments (e.g., 2.18 GB vs 2.42 GB for ResNet-50 to ViT-B-16).
• On GSM8K, PRT trained with Qwen2-0.5B and applied to Qwen2.5-7B reaches 70.7 versus 9.5 pretrained and 41.8 EFT (Table of upgraded versions).
• Reported that PRT achieves accuracy comparable to EFT with less inference cost; training overhead is slightly higher (e.g., ViT-B-16 batch time 116 ms FT vs 151 ms PRT).

Measured effects reported in the paper

Surprising findings

• PRT did not improve HumanEval for Falcon3 at larger target sizes, performing below the pretrained model, because the source 1B model had little code ability.
• Entropy regularization of the reward did not degrade PRT performance even as reward accuracy dropped, and sometimes boosted it.

Limitations

• Requires the new pretrained model to share the same label space or vocabulary as the source model.
• Gains are inconsistent across tasks; e.g., HumanEval degrades for Falcon3 when the source model lacks the capability.
• Optimal entropy regularization coefficient depends on task and pretrained model and is not yet practical.

Experimental setup

Datasets: Tulu v2 Cars CUB Aircraft Caltech101 CIFAR-100 Country211 Flowers RESISC45

Benchmarks: GSM8K IFEval HumanEval image classification test sets (Cars, CUB, Aircraft, Caltech101, CIFAR-100, Country211, Flowers, RESISC45)

Metrics: accuracy exact match IFEval average of four scores pass@1 peak memory time per token/batch

Baselines: Emulated fine-tuning (EFT) Pretrained (zero-shot) Fine-tuned / instruct-tuned oracle

Builds on: emulated fine-tuning proxy-tuning KL-regularized reward maximization PAC-Bayes

Closest papers by meaning

• Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections (NeurIPS 2025 · similarity 0.89)
• GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.89)
• Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models (NeurIPS 2026 · similarity 0.89)
• Scaling Reward Modeling without Human Supervision (NeurIPS 2026 · similarity 0.88)
• Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.88)
• ToolRL: Reward is All Tool Learning Needs (NeurIPS 2025 · similarity 0.88)