Research map / Large language models / Alignment and preferences

Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment

AAAI 2025 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Formulates LLM alignment via a trainable prompter that rewrites prompts for a frozen LLM, derives a closed-form optimal prompt distribution and a suboptimality bound versus RLHF, and validates with PPO-trained prompters on three datasets.

Problem

Aligning LLMs when the target model is frozen or black-box, so parameter fine-tuning (RLHF) is infeasible; the theoretical relationship between prompt optimization and fine-tuned alignment is unexplored.

Approach

Treat a small prompter model as a KL-regularized policy over rewritten prompts fed to a frozen LLM; derive the closed-form optimal prompter rho* (softmax of reward over prompts weighted by the SFT prompter) and bound its suboptimality gap relative to the RLHF-optimal policy via TV distances and a KL term controlled by lambda.

Key results

• Align-Pro win rates against no fine-tuning ranged from 46% to 67% (Align-Pro vs. baseline) across datasets and model pairs, with the baseline winning as low as 23%; for example 65 vs 23 on UltraFeedback with Qwen-2.5-1.5B prompter and Llama-3.1-8B frozen model.
• Align-Pro reported the lowest reward variance and mean reward above the no fine-tuning baseline, approaching the RLHF oracle, per the paper's claims.
• Theory: the suboptimality bound is non-negative only when the KL between rho* and rho_sft is at most (eps1+eps2)/lambda.

Measured effects reported in the paper

Surprising findings

• The paper reports Align-Pro has the lowest reward variance, which it attributes to less diverse responses from precise prompting.
• In one configuration (Phi-3.5 prompter, Qwen-2.5-7B frozen, Orca), win rates were tied at 46 vs 46.

Limitations

• Performance is bounded by the capabilities of the frozen LLM and is sensitive to small prompt changes.
• Only the upper bound is derived; no lower bounds on suboptimality; no comparison against other prompt optimization methods (by design).

Experimental setup

Datasets: UltraFeedback HelpSteer Orca

Benchmarks: UltraFeedback HelpSteer Orca

Metrics: mean reward reward variance win rate

Baselines: No Fine-Tuning RLHF

Builds on: RLHF direct preference optimization proximal policy optimization Bradley-Terry model

Closest papers by meaning

• Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.93)
• Preference Ranking Optimization for Human Alignment (AAAI 2024 · similarity 0.92)
• Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.91)
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.91)
• Alignment of Large Language Models with Constrained Learning (NeurIPS 2025 · similarity 0.91)
• Aligning Large Language Models with Representation Editing: A Control Perspective (NeurIPS 2024 · similarity 0.91)