Research map / Large language models / Instruction and fine-tuning
L3Ms — Lagrange Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formulates SFT and alignment of LLMs as a constrained optimization problem and solves it with logarithmic barriers (L3Ms), fine-tuning LLaMA-7B on UltraChat with length and helpful/harmless constraints.
Problem
Current LLM pipelines combine SFT and alignment with heuristically chosen reward weights and KL penalties, making it hard to enforce application-specific minimum requirements on preferences while preserving task performance.
Approach
Merge SFT and alignment into one constrained problem (minimize task NLL subject to average preference-reward constraints) and enforce constraints with a relaxed logarithmic barrier whose parameter mu is annealed during fine-tuning, avoiding learned Lagrange multipliers and deviation penalties.
Key results
- L3Ms satisfy length constraints with mean response lengths in the required range, with task perplexity within 0.02 standard deviations of the SFT model.
- On helpful-harmless constraints, L3Ms reach mean task perplexities of 0.804-0.820 versus MMs at 0.822 or higher, while satisfying the reward constraints.
- SFT model task perplexity is 0.805 (+/-0.3) versus 1.316 at initialization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use log-barrier constrained fine-tuning instead of minimax optimization | LLaMA-7B on UltraChat with helpfulness and harmlessness constraints · UltraChat held-out test | task perplexity: MMs: mean task perplexities ≥ 0.822; L3Ms: 0.804-0.820 | Helped |
| Add response-length constraints | L3M · UltraChat held-out test, varying response-length constraints | task perplexity: within 0.02 standard deviations | No clear effect |
| Make response-length constraints more stringent | L3M · UltraChat held-out test, varying response-length constraints | task perplexity | Hurt |
| Fine-tune on UltraChat with supervised fine-tuning | LLaMA-7B initialization · UltraChat held-out test | task perplexity: 1.316\textsubscript{$\pm$0.4} -> 0.805\textsubscript{$\pm$0.3} | Helped |
Surprising findings
- Naively averaging perfectly anti-correlated length rewards makes unconstrained alignment, including RLHF, ineffective because the loss vanishes.
Limitations
- Convergence of the procedure is not guaranteed in this non-convex setting.
- Constraints are enforced only on average (expectation), not uniformly or via chance constraints.
- Experiments use a single base model (LLaMA-7B) and evaluate mainly on constraint satisfaction and perplexity rather than downstream quality.
Experimental setup
Datasets: UltraChat Helpful and Harmless (HH)
Benchmarks: UltraChat held-out test set Helpful and Harmless preference test set
Metrics: task perplexity response length distribution mean helpfulness/harmlessness reward
Baselines: SFT model MMs (minimax Lagrangian approach) unconstrained reward-weighted alignment
Builds on: minimax Lagrangian constrained RLHF Bradley-Terry reward modeling interior-point log barrier methods
Closest papers by meaning
- Alignment of Large Language Models with Constrained Learning (NeurIPS 2025 · similarity 0.91)
- Aligning Large Language Models with Representation Editing: A Control Perspective (NeurIPS 2024 · similarity 0.89)
- Long-Short Alignment for Effective Long-Context Modeling in LLMs (ICML 2025 · similarity 0.89)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.89)
- Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach (ICML 2025 · similarity 0.89)
- Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.