Research map / Bandits and online learning / Contextual bandits and Thompson sampling
Risk-Aware Continuous Control with Neural Contextual Bandits
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a risk-aware actor multi-critic framework using quantile-based distributional critics for constrained continuous-action contextual bandits, with a risk parameter alpha that trades off constraint satisfaction against reward, evaluated on synthetic and 5G O-RAN experiments.
Problem
Contextual bandits with stepwise stochastic constraints and continuous action spaces, where aleatoric noise in reward and constraint metrics makes constraint violations likely if only mean values are used.
Approach
Use one distributional (quantile) critic per metric, aggregate the reward mean with a penalty on the alpha-quantile of each constraint critic, and train a deterministic actor conditioned on the risk level alpha so that risk can be tuned at inference without new data.
Key results
- In the 5G testbed, the proposed method consistently satisfies the reliability constraint with near-zero unreliability, at only 8.5% higher power consumption than MC-NCB on average.
- In the synthetic environment, the method with alpha=0.995 gives the lowest constraint violation across noise levels, while alpha=0.5 gives the highest reward with more violations.
- SafeOPT's per-GP hyperparameter optimization takes about 160.7 s and its runtime grows exponentially with sample size, while the proposed inference takes about 0.106 ms.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use risk-aware distributional critics and a risk-conditioned actor | constrained contextual bandit · Synthetic environment | accumulated constraint violation | Helped |
| Increase risk level α from 0.5 to 0.995 (ablation) | Risk-Aware Neural Contextual Bandit · Synthetic environment | reward | Hurt |
| Increase risk level α (ablation) | Risk-Aware Neural Contextual Bandit · Synthetic environment inference | constraint violation | Helped |
| Increase risk level α (ablation) | Risk-Aware Neural Contextual Bandit · Synthetic environment inference | reward | Hurt |
| Use distributional critic instead of a mean-value critic (ablation) | NCB · Synthetic environment | constraint violation | Mixed |
| Use separate critics for reward and constraints instead of one utility critic (ablation) | NCB · Synthetic environment | constraint violation | Mixed |
| Set λ above 2.5 (ablation) | Risk-Aware Neural Contextual Bandit · Synthetic environment, σ_env = 0.2 | constraint violation | No clear effect |
| Set λ above 2.5 (ablation) | Risk-Aware Neural Contextual Bandit · Synthetic environment, σ_env = 0.2 | reward | Hurt |
| Increase context and action dimensionality (ablation) | Risk-Aware Neural Contextual Bandit · Synthetic dimensionality environment | constraint satisfaction during training | Helped |
| Use Risk-Aware Neural Contextual Bandit instead of the benchmarks | 5G mobile network resource assignment · 5G mobile network inference | reliability | Helped |
| Use Risk-Aware Neural Contextual Bandit instead of MC-NCB | 5G mobile network resource assignment · 5G mobile network inference | power consumption: 8.5\% | Hurt |
| Use SafeOPT instead of the neural method | contextual bandit inference · Synthetic environment | execution time | Hurt |
Surprising findings
- SafeOPT could not be evaluated in the 5G use case because its inference time exceeded the 100 ms decision period.
Limitations
- GP-based SafeOPT requires O(n^3) computation and domain knowledge for an initial safe set, which the authors treat as a drawback of the baseline rather than a limitation of their own method.
- Comparison to SafeOPT in training is not fair due to its prior knowledge requirements, so training curves exclude it.
- Evaluation uses a single 5G testbed and a synthetic environment with simple quadratic functions.
Experimental setup
Datasets: synthetic quadratic contextual bandit environment real 5G BS traces from Madrid (May 2023), emulated via Falcon
Benchmarks: synthetic quadratic contextual bandit environment 5G mobile network resource assignment testbed
Metrics: accumulated constraint violation reward unreliability power consumption inference time
Baselines: NCB SC-DNCB MC-NCB SafeOPT
Builds on: DDPG SafeOPT quantile Huber loss
Closest papers by meaning
- Safety-Aware Algorithms for Adversarial Contextual Bandit (ICML 2017 · similarity 0.89)
- Conservative Contextual Bandits: Beyond Linear Representations (ICLR 2025 · similarity 0.87)
- Policy Optimization for CMDPs with Bandit Feedback: Learning Stochastic and Adversarial Constraints (ICML 2025 · similarity 0.86)
- Learning Safe Control via On-the-Fly Bandit Exploration (ICML 2025 · similarity 0.86)
- Contextual Linear Optimization with Bandit Feedback (NeurIPS 2024 · similarity 0.85)
- Cost-Aware Near-Optimal Policy Learning (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.