Research map / Large language models / Alignment and preferences
Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SITAlign, an inference-time constrained decoding method that maximizes a primary reward while enforcing threshold constraints on secondary rewards via Lagrangian duality and Transfer Q* estimates.
Problem
Multi-objective LLM alignment usually maximizes a weighted sum of rewards, which ignores that humans satisfice: they maximize a primary objective while keeping other attributes above acceptable thresholds. Existing inference-time methods also need weights or finetuning.
Approach
Cast inference-time decoding as a constrained problem: maximize the primary reward's Q-value under KL regularization, subject to expected secondary-reward Q-values exceeding thresholds. Solve it via Lagrangian duality with a closed-form, quadratic-approximated dual estimate and Transfer Q* as the Q-function estimate, yielding a token-level reweighting of the baseline policy.
Key results
- On PKU-SafeRLHF (helpfulness primary, harmlessness threshold), SITAlign improves helpfulness GPT-4 win-tie rate by 22.3% over MOD while meeting the 50% harmlessness threshold.
- On the Helpfulness-Humor setup, SITAlign improves the win-tie rate by about 10% over the state-of-the-art MOD baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SITAlign instead of MOD to maximize helpfulness while meeting the harmlessness threshold | Zephyr-7B-β · PKU-SafeRLHF | helpfulness GPT-4 win-tie rate: 22.3% | Helped |
| Use SITAlign instead of MOD while meeting the humor threshold | MPT-7B-Chat · Anthropic-HH | GPT-4 win-tie rate: approximately $10\%$ | Helped |
| Use SITAlign instead of MOD to maximize summary quality while meeting the faithfulness threshold | Minotaur-7B · Summarize-from-Feedback | summary-quality GPT-4 win-tie rate | Helped |
| Use MOD instead of SITAlign to prioritize the target reward | SITAlign · All three evaluation setups | target-reward GPT-4 win-tie rate | Hurt |
| Increase the threshold β (ablation) | SITAlign · Evaluation-1 and Evaluation-2 | GPT-4 win-tie rate | Mixed |
| Raise the target-reward threshold beyond the point where its win-rate stabilizes (ablation) | SITAlign · Evaluation-1 and Evaluation-2 | target-reward GPT-4 win-tie rate | No clear effect |
Surprising findings
- Responses with reward above about -12 were judged harmless by GPT-4 in roughly 90% of cases, supporting the idea that thresholds suffice instead of maximizing the target reward.
- MOD achieves higher target-reward win rates but performs worse on the baseline reward, suggesting maximizing all objectives is not always best.
Limitations
- The suboptimality bounds rely on assumptions such as known optimal dual variables, and the dual is approximated by a quadratic approximation.
- Evaluation relies on GPT-4 win-tie rates as a proxy for human judgment, with no human evaluation reported.
- Threshold selection requires tuning (e.g., -12 for harmlessness, -0.5 for humor) and is dataset-specific.
Experimental setup
Datasets: PKU-SafeRLHF-30K Anthropic-HH Summarize-from-Feedback
Benchmarks: PKU-SafeRLHF test prompts Anthropic-HH Summarize-from-Feedback
Metrics: GPT-4 win-tie rate on baseline reward GPT-4 win-tie rate on target reward
Baselines: MOD Best-of-K Zephyr-7B-beta baseline decoding helpful policy harmless policy
Builds on: Transfer Q* controlled decoding multi-objective decoding (MOD) KL-regularized RLHF
Closest papers by meaning
- On-the-fly Preference Alignment via Principle-Guided Decoding (ICLR 2025 · similarity 0.90)
- Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.90)
- Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.89)
- Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.89)
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.89)
- Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.