Research map / Reinforcement learning / Policy gradient and actor-critic
LipsNet++: Unifying Filter and Controller into a Policy Network
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LipsNet++, a policy network combining a trainable Fourier filter layer (to suppress observation noise) and a Lipschitz controller layer trained with Jacobian regularization (to smooth the policy), evaluated for action smoothness and noise robustness in double integrator, DMControl, and mini-vehicle experiments.
Problem
Reinforcement learning policies suffer from action fluctuation under observation noise and non-smooth policy functions, causing actuator wear, safety risk, and performance loss in real-world control.
Approach
Decouples the two causes of action fluctuation: a learnable complex filter matrix in the frequency domain (FFT/IFFT) removes noise frequencies from observation history, and a network-wise Jacobian norm regularizer in the actor loss keeps the policy Lipschitz-smooth without needing Jacobians at inference time.
Key results
- In DMControl Walker with noise, LipsNet++ increases TAR by 3.4% and reduces AFR by 35.5% versus LipsNet.
- In double integrator at noise amplitude 0.3, LipsNet++ gets 8.2% higher TAR and 75.0% lower AFR than LipsNet-L.
- On a physical mini-vehicle, LipsNet++ increases TAR by 5.8% and reduces AFR by 90.0% compared to MLP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LipsNet++ instead of LipsNet | DMControl Walker noisy · Walker | TAR: +3.4% | Helped |
| Use LipsNet++ instead of LipsNet | DMControl Walker noisy · Walker | AFR: -35.5% | Helped |
| Use LipsNet++ instead of MLP | Mini-vehicle policy · Mini-vehicle scenarios 1–3, noise amplitude 20 | TAR: +5.9% | Helped |
| Use LipsNet++ instead of MLP | Mini-vehicle policy · Mini-vehicle scenarios 1–3, noise amplitude 20 | AFR: -90.0% | Helped |
| Use LipsNet++ instead of LipsNet-L | Double integrator policy · Double integrator, observation noise amplitude 0.3 | TAR: +8.2% | Helped |
| Use LipsNet++ instead of LipsNet-L | Double integrator policy · Double integrator, observation noise amplitude 0.3 | AFR: -75.0% | Helped |
Limitations
- Jacobian regularization and filter penalty add hyperparameters (lambda_k, lambda_h, N) requiring tuning per environment
- Evaluation limited to a double integrator, four DMControl tasks, and a mini-vehicle
Experimental setup
Datasets: DeepMind Control Suite (Cartpole, Reacher, Cheetah, Walker) double integrator simulation mini-vehicle trajectory tracking
Benchmarks: double integrator DeepMind Control Suite mini-vehicle driving
Metrics: total average return (TAR) action fluctuation ratio (AFR) forward/backward speed
Baselines: MLP MLP-SN LipsNet-G LipsNet-L CAPS L2C2
Builds on: LipsNet MLP-SN TD3 DSAC INFADP
Closest papers by meaning
- LipsNet: A Smooth and Robust Neural Network with Adaptive Lipschitz Constant for High Accuracy Optimal Control (ICML 2023 · similarity 0.94)
- ODE-based Smoothing Neural Network for Reinforcement Learning Tasks (ICLR 2025 · similarity 0.89)
- Towards Generalization and Simplicity in Continuous Control (NeurIPS 2017 · similarity 0.88)
- Improve Robustness of Reinforcement Learning against Observation Perturbations via l∞ Lipschitz Policy Networks (AAAI 2024 · similarity 0.88)
- Analyzing Generalization in Policy Networks: A Case Study with the Double-Integrator System (AAAI 2024 · similarity 0.85)
- No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learning (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.