Research map / Reinforcement learning / RL from feedback
Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZO-RankSGD, a zeroth-order optimizer that uses only (m,k)-ranking oracle feedback, with a convergence guarantee, and applies it to RL policy search and human-in-the-loop Stable Diffusion latent optimization.
Problem
Optimizing a black-box objective that can only be accessed through a ranking oracle (e.g., human judges ranking outputs), where exact function values are unavailable.
Approach
A rank-based random gradient estimator that turns the sorted top-k output of an (m,k)-ranking oracle into a DAG of pairwise comparisons and a weighted sum of Gaussian perturbations; the analysis gives a stationary-point convergence rate and variance terms that depend on m and k.
Key results
- On quadratic and Rosenbrock functions (d=100), ZO-RankSGD outperforms GLD-Fast, CMA-ES and SCOBO and performs similarly to ZO-SGD, which uses exact function values.
- On MuJoCo tasks with a (5,5)-ranking oracle, ZO-RankSGD outperforms CMA-ES by a significant margin on Reacher, Swimmer and HalfCheetah.
- For Stable Diffusion, optimizing the latent embedding with human ranking feedback improves detail (e.g., fingers, eyes) within only a few rounds of feedback; qualitative results only, no quantitative metric reported.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ZO-RankSGD instead of GLD-Fast or CMA-ES | zeroth-order optimization · Quadratic function and Rosenbrock function | objective value | Helped |
| Use ZO-RankSGD instead of SCOBO | zeroth-order optimization · Quadratic function and Rosenbrock function | objective value | Helped |
| Use ranking feedback with ZO-RankSGD instead of value feedback with ZO-SGD | zeroth-order optimization · Quadratic function and Rosenbrock function | objective value | No clear effect |
| Use ZO-RankSGD instead of CMA-ES | MuJoCo policy optimization · Reacher-v2, Swimmer-v2, and HalfCheetah-v2 | episode reward | Helped |
| Use ranking feedback with ZO-RankSGD instead of value feedback with ZO-SGD | MuJoCo policy optimization · Reacher-v2, Swimmer-v2, and HalfCheetah-v2 | episode reward | No clear effect |
| Use full ranking information with m = k rather than top-1 ranking with k = 1 (ablation) | ZO-RankSGD · Quadratic function and Rosenbrock function | optimization performance | Helped |
| Optimize latent noise embeddings using human ranking feedback | Stable Diffusion · Text-to-image generation | image detail and realism | Helped |
| Optimize images using CLIP similarity instead of human preference | Stable Diffusion · Text-to-image generation | human preference | Hurt |
Surprising findings
- Ranking-only feedback performs almost as well as value-oracle ZO-SGD in the reported experiments.
Limitations
- Human feedback experiments were produced by the authors themselves, with no large-scale user study.
- Noisy or inconsistent ranking feedback is only treated empirically and lacks theory.
- Convergence is to a stationary point, and the variance bound for non-quadratic functions relies on the non-diminishing M1 term.
Experimental setup
Datasets: MuJoCo Reacher-v2 MuJoCo Swimmer-v2 MuJoCo HalfCheetah-v2
Benchmarks: Quadratic function (d=100) Rosenbrock function (d=100) MuJoCo Reacher-v2 MuJoCo Swimmer-v2 MuJoCo HalfCheetah-v2
Metrics: gradient norm function value episode reward
Baselines: ZO-SGD SCOBO GLD-Fast CMA-ES
Builds on: ZO-SGD comparison-based gradient estimator with one-bit compressive sensing CMA-ES DDIM DPM-solver
Closest papers by meaning
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.90)
- Adaptive Preference Scaling for Reinforcement Learning with Human Feedback (NeurIPS 2024 · similarity 0.87)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.87)
- Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF (ICML 2025 · similarity 0.87)
- Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.87)
- DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.