Research map / Reinforcement learning / Offline RL

Low-Rank Adaptation for Critic Learning in Off-Policy Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › Offline RL · approach: Offline RL sample complexity theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes using LoRA with frozen random base matrices as a structural regularizer for off-policy critics (SimbaV2, BRC), with a hyperspherical-compatible variant, showing gains over dense and random-sparse critics on DMC-Hard and IsaacLab.

Problem

Large off-policy critics overfit and become unstable under replay-based bootstrapped training, so scaling critic capacity does not reliably help.

Approach

Freeze randomly initialized dense critic weight matrices and train only low-rank LoRA adapters in critic residual blocks, keeping the forward pass dense while constraining updates to a low-dimensional subspace; a LoRA-compatible hyperspherical projection solves a per-row scalar so effective weights stay on the unit sphere.</alignment>

Key results

• On DMC-Hard with SAC+SimbaV2 (d_h=512, L=2), LoRA averages 701.3 return vs 701.1 for dense SimbaV2 and 607.3 for Sparse.
• LoRA gives the best return on most IsaacLab FastTD3+SimbaV2 tasks and on most locomotion tasks, with critic loss generally lower than dense and sparse variants.
• In a toy chain MDP, small-rank LoRA (r=1) gives lower final true-Q error than the dense critic under bootstrapped off-policy TD, while dense is best under static regression.

Surprising findings

• Under static regression the dense critic is best, but under bootstrapped off-policy TD small-rank LoRA beats it, suggesting the low-rank constraint acts as an optimization regularizer rather than a capacity limit.
• Dense warm-up before switching to LoRA does not systematically outperform pure LoRA from scratch.

Limitations

• Gains on some tasks are within standard deviation, and results are reported as 'best or competitive' rather than uniformly best.
• The SimbaV2-compatible projection requires a base-norm choice (0.5) and nonzero LoRA initialization, adding design constraints.
• Evaluation is limited to DMC-Hard locomotion and six IsaacLab tasks with 5 seeds.

Experimental setup

Datasets: DMC-Hard IsaacLab

Benchmarks: DMC-Hard IsaacLab

Metrics: average normalized return final episode return critic loss

Baselines: SimbaV2 BRC Sparse (one-shot random pruning) FastTD3

Builds on: LoRA SimbaV2 BRC Soft Actor-Critic FastTD3

Closest papers by meaning

• Model-free Low-Rank Reinforcement Learning via Leveraged Entry-wise Matrix Estimation (NeurIPS 2024 · similarity 0.89)
• Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning (ICML 2024 · similarity 0.89)
• Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.87)
• Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.87)
• Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training (NeurIPS 2026 · similarity 0.87)
• Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.87)