Research map / Optimization / Optimizers
Stochastic Optimal Control and Estimation with Multiplicative and Internal Noise
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that the widely used LQMG optimal control algorithm of Todorov and Li (TOD) relies on an invalid unbiasedness assumption, and proposes gradient-descent (GD) and analytic fixed-point (FPOMP) solvers based on moment propagation that achieve lower cost, especially with internal noise.
Problem
Optimal feedback control under a realistic sensorimotor noise model (multiplicative motor and sensory noise plus internal estimation noise) is hard; the existing state-of-the-art algorithm assumes unbiased estimators, which does not hold, leading to suboptimal control and filter gains.
Approach
Treat the expected total cost as a function of linear control gains and Kalman-like filter gains, computed in closed form by propagating the first and second moments of state and estimate, then minimize it by gradient descent (GD) or via an analytic backward/forward fixed-point iteration (FPOMP) without assuming unbiasedness.
Key results
- In a 1D reaching task with internal noise, GD and FPOMP achieve lower expected cost than TOD across internal noise levels; in the 4-D single-joint reaching task, GD reduces cost by up to 90% when internal noise is about 10% of total noise (sigma_eta=0.05).
- The unbiasedness condition E[x_t|x_hat_t]=x_hat_t used by TOD is shown to be violated numerically, and it implies the orthogonality principle, which fails for the optimal filter when internal noise is nonzero.
- FPOMP, the analytic fixed-point counterpart, matches GD solutions and runtime is comparable to TOD, while GD takes about 4 hours versus minutes for TOD on the 4-D task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize control and filter gains by minimizing expected cost without assuming unbiased estimation, instead of using TOD | LQMG sensorimotor control · Single-joint reaching task | expected accumulated cost: reducing the cost by up to 90% when internal noise contributes approximately 10% of the total noise (ση = 0.05) | Helped |
| Increase internal noise | LQMG control solved by GD versus TOD · One-dimensional reaching task | expected accumulated cost | Mixed |
| Use FPOMP, the analytical counterpart to numerical GD | LQMG control · One-dimensional reaching task | expected accumulated cost | No clear effect |
| Optimize gains without assuming unbiasedness when filter gains are fixed at suboptimal values | LQMG control · One-dimensional problem with zero internal noise | expected accumulated cost | Helped |
Surprising findings
- Optimal control gains decrease as internal noise increases, yielding smoother control and slower movements with reduced trial-to-trial variability.
- The TOD solution is suboptimal even without internal noise when the algorithm has not converged or when filter/control gains are fixed suboptimally.
Limitations
- Filter gains are assumed state-independent (non-adaptive), which is suboptimal under multiplicative noise.
- Global convergence is not proven; only local minima guarantee, though none were found numerically.
- FPOMP is derived only for the LQAG case with internal noise; the full multiplicative-noise case is left for future work.
Experimental setup
Metrics: expected accumulated cost E[J] control gain magnitude trial-to-trial variability estimation bias E[x|x_hat]
Baselines: TOD (Todorov and Li 2005 algorithm)
Builds on: Todorov and Li iterative LQG with multiplicative noise (TOD) linear-quadratic-Gaussian control Kalman filter
Closest papers by meaning
- Falcon: Fast Visuomotor Policies via Partial Denoising (ICML 2025 · similarity 0.85)
- Memory-Augmented Potential Field Theory: A Framework for Adaptive Control in Non-Convex Domains (NeurIPS 2025 · similarity 0.84)
- Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence (ICML 2025 · similarity 0.84)
- Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States (ICML 2024 · similarity 0.84)
- Stochastic Optimal Control Matching (NeurIPS 2024 · similarity 0.84)
- Inverse decision-making using neural amortized Bayesian actors (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.