Research map / Reinforcement learning / Policy gradient and actor-critic
Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MOAC, an MGDA-style actor-critic for multi-objective RL with finite-time Pareto-stationary convergence and O(ε^-2 log ε^-1) sample complexity independent of the number of objectives, using a momentum-weighted common descent direction.
Problem
Multi-objective reinforcement learning lacks finite-time convergence and sample complexity guarantees; actor-critic methods face coupling between actor and critic and biased stochastic MGDA-type gradient estimates.
Approach
MOAC runs an actor-critic with per-objective TD-error policy gradients, combines them via a QP-derived common descent direction smoothed by a momentum coefficient on the weights, which controls cumulative bias and makes rates independent of the number of objectives; the momentum schedule also initializes weights from environment samples.
Key results
- Theory: MOAC attains E||∇J λ||^2 ≤ ε + O(ζ_approx) with total sample complexity O(ε^-2 log ε^-1), independent of the number of objectives M.
- On the KuaiRand dataset, MOAC improves Like by 6.57%, Follow by 10.0% and Comment by 1.27% over Behavior-Clone, outperforming PDPG and TSCAC on most objectives.
- Versus fixed-weight variant, MOAC gains are reported on most objectives (e.g., Like 1.312 vs 1.287), with Dislike the exception.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace dynamically updated momentum gradient weights with fixed weights initialized in the first iteration (ablation) | MOAC · Kuaishou recommendation logs | Click: 0.539 -> 0.541 | Helped |
| replace dynamically updated momentum gradient weights with fixed weights initialized in the first iteration (ablation) | MOAC · Kuaishou recommendation logs | Like: 1.287 -> 1.312 | Helped |
| replace dynamically updated momentum gradient weights with fixed weights initialized in the first iteration (ablation) | MOAC · Kuaishou recommendation logs | Follow: 4.800 -> 5.070 | Helped |
| replace dynamically updated momentum gradient weights with fixed weights initialized in the first iteration (ablation) | MOAC · Kuaishou recommendation logs | Comment: 3.151 -> 3.266 | Helped |
| replace dynamically updated momentum gradient weights with fixed weights initialized in the first iteration (ablation) | MOAC · Kuaishou recommendation logs | Forward: 0.897 -> 1.066 | Helped |
| replace dynamically updated momentum gradient weights with fixed weights initialized in the first iteration (ablation) | MOAC · Kuaishou recommendation logs | Dislike: 1.428 -> 1.486 | Hurt |
| replace dynamically updated momentum gradient weights with fixed weights initialized in the first iteration (ablation) | MOAC · Kuaishou recommendation logs | WatchTime: 1.282 -> 1.307 | Helped |
Surprising findings
- TSCAC beats MOAC on Click, Dislike and WatchTime, and MOAC is worse than Behavior-Clone on Forward.
- The fixed-weight variant outperforms MOAC on Dislike (1.428 vs 1.486 is in MOAC's favour? no; MOAC's Dislike 1.486 is worse than fixed-weight 1.428), indicating the dynamic weighting does not help every objective.
Limitations
- Assumes linear value function approximation and finite state/action spaces.
- Guarantees are to an epsilon-Pareto-stationary point, not Pareto optimality; the momentum choice trades off direction optimality against convergence rate.
- Real-world experiments are off-policy adaptations on one dataset with imbalanced rewards and do not beat all baselines.
Experimental setup
Datasets: KuaiRand (Kuaishou recommendation logs) MO-Gymnasium resource-gathering-v0 MO-Gymnasium fishwood-v0
Benchmarks: KuaiRand (Kuaishou logs, NCIS evaluation) MO-Gymnasium resource-gathering-v0 MO-Gymnasium fishwood-v0
Metrics: normalised capped importance sampling (NCIS) score discounted reward per objective squared l2-norm of policy gradient
Baselines: Behavior-Clone TSCAC PDPG fixed-weight MOAC
Builds on: multi-gradient descent algorithm (MGDA) PDPG TSCAC
Closest papers by meaning
- Actor-Critics Can Achieve Optimal Sample Efficiency (ICML 2025 · similarity 0.89)
- Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximation (ICML 2024 · similarity 0.89)
- Multi-Objective Reinforcement Learning with Max-Min Criterion: A Game-Theoretic Approach (NeurIPS 2025 · similarity 0.88)
- Towards Global Optimality for Practical Average Reward Reinforcement Learning without Mixing Time Oracles (ICML 2024 · similarity 0.88)
- Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization (NeurIPS 2026 · similarity 0.88)
- On Generalization Across Environments In Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.