Research map / Learning theory and algorithms / Prediction and decision losses
Beyond Value Functions: Single-Loop Bilevel Optimization under Flatness Conditions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PBGD-Free, a fully single-loop, value-function-free first-order bilevel method, and shows convergence under a relaxed (δ,α)-flatness condition for LLM PEFT with DPO upper and SFT lower objectives.
Problem
Bilevel optimization for LLM fine-tuning is costly because existing first-order penalty methods need nested inner loops (value-function estimation) and Lipschitz assumptions on the upper-level objective fail in practice.
Approach
Drop the value-function gradient term from the penalty-based update (single-loop, one inner step per iteration) and prove convergence by replacing the uniform Lipschitz assumption on the upper-level objective with a Hölder-like (δ,α)-flatness condition that holds near the lower-level solution in representation-learning PEFT.
Key results
- On Llama-3-3b PEFT, PBGD-Free reaches DPO eval loss 0.7837 and SFT eval loss 0.8516, lower than V-PBGD (0.818, 1.0309), BOME (0.8332, 1.1552) and ALRIGHT (0.8055, 0.8656).
- Theory: without the value-function term, PBGD-Free has a non-vanishing Theta(l_{f,0}^2) gradient residual under Lipschitz assumption; under (delta,alpha)-flatness with small delta it converges at O(T^-1 + delta^{2(alpha-1)/alpha}).
- Ablation on Pythia-1b suggests gamma around 10 balances DPO and SFT losses.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PBGD-Free instead of the strongest tabulated baseline, ALRIGHT | Llama-3-3b PEFT · Llama-3-3b PEFT, outcome (x_T,y_T) | DPO eval loss: 0.8055 -> 0.7837 | Helped |
| Use PBGD-Free instead of the strongest tabulated baseline, ALRIGHT | Llama-3-3b PEFT · Llama-3-3b PEFT, outcome (x_T,y_T) | SFT eval loss: 0.8656 -> 0.8516 | Helped |
| Use PBGD-Free instead of the strongest tabulated baseline, ALRIGHT | Llama-3-3b PEFT · Llama-3-3b PEFT, post-SFT outcome (x_T, ỹ) | DPO eval loss: 0.8201 -> 0.8088 | Helped |
| Use PBGD-Free instead of the strongest tabulated baseline, ALRIGHT | Llama-3-3b PEFT · Llama-3-3b PEFT, post-SFT outcome (x_T, ỹ) | SFT train loss: 0.7855 -> 0.6688 | Helped |
| Increase penalty constant γ (ablation) | PBGD-Free on Pythia-1b · BLO PEFT learning phase (S1) | DPO loss | Hurt |
| Increase penalty constant γ (ablation) | PBGD-Free on Pythia-1b · BLO PEFT learning phase (S1) | SFT loss | Helped |
| Remove value-function gradient and use the PBGD-Free update | PBGD under the general Lipschitz condition · Bilevel optimization counterexample | stationarity residual: ${\Theta}(l_{f,0}^2)$ | Hurt |
Surprising findings
- A simple counterexample shows that dropping the value-function gradient term can converge to a wrong point under the Lipschitz assumption, with a residual of order l_{f,0}^2.
- The PBGD-Free method achieved lower loss than the more expensive double-loop baselines on the LLM task despite the approximation.
Limitations
- Convergence guarantee relies on a flatness constant delta(x) being small along the trajectory, which is assumed rather than universally verified.
- Under the plain Lipschitz condition, the method only reaches a Theta(1) neighborhood of a stationary point.
- Theory requires PL condition on the lower level and alpha in (1,1.5] for the main results.
Experimental setup
Datasets: Dahoas/rm-hh-rlhf OpenOrca NLSY-7k
Benchmarks: Dahoas/rm-hh-rlhf OpenOrca
Metrics: DPO loss SFT loss BLEU-4 corpus score BLEU-4 sentence score training loss vs time
Baselines: V-PBGD BOME F2SA ALRIGHT BVFSM F2SA-MA
Builds on: penalty-based bilevel gradient descent (PBGD) F2SA DPO LoRA
Closest papers by meaning
- Tuning-Free Bilevel Optimization: New Algorithms and Convergence Analysis (ICLR 2025 · similarity 0.89)
- A Nearly Optimal Single Loop Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness (ICML 2024 · similarity 0.89)
- Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis (ICLR 2024 · similarity 0.87)
- Double Momentum Method for Lower-Level Constrained Bilevel Optimization (ICML 2024 · similarity 0.86)
- qNBO: quasi-Newton Meets Bilevel Optimization (ICLR 2025 · similarity 0.86)
- Generalized Smooth Bilevel Optimization with Nonconvex Lower-Level (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.