Research map / Reinforcement learning / MDPs and planning
AoI-MDP: An AoI Optimized Markov Decision Process Dedicated in the Underwater Task (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AoI-MDP, an MDP extension for underwater AUV data collection that adds observation delay to the state, wait time to the action, and time-averaged age of information to the reward, trained with RL.
Problem
Autonomous underwater vehicles relying on RL suffer from observation delay due to acoustic communication, and standard MDPs assume instant state information, which reduces information freshness and decision-making efficiency.
Approach
Model observation delay as acoustic signal delay estimated via statistical signal processing and include it in the state; add a wait-time component to the action space; and add negative time-averaged AoI as an extra reward component so RL jointly optimizes information freshness and decisions.
Key results
- AoI-MDP achieves lower time-averaged AoI, lower energy consumption, higher sum data rate and higher cumulative reward than standard MDP over 250 training epochs (Figure 3).
- Under delay models, AoI-MDP with SSP delay reaches AoI 1.97±0.26, SDR 11.99±0.73, EC 33.83±2.59, versus Poisson AoI 3.42±0.18 and Exponential AoI 2.67±0.26.
- The SSP-based delay model reports near-optimal AoI, data rate and energy consumption in the data collection task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AoI-MDP instead of standard MDP | RL training for the underwater data collection task · Underwater data collection task | time-averaged AoI | Helped |
| Use AoI-MDP instead of standard MDP | RL training for the underwater data collection task · Underwater data collection task | energy consumption | Helped |
| Use AoI-MDP instead of standard MDP | RL training for the underwater data collection task · Underwater data collection task | sum data rate | Helped |
| Use AoI-MDP instead of standard MDP | RL training for the underwater data collection task · Underwater data collection task | cumulative reward | Helped |
| Use the SSP delay model instead of a Poisson delay model | AoI-MDP · Underwater data collection task | AoI: 3.42±0.18 -> 1.97±0.26 | Helped |
| Use the SSP delay model instead of an exponential delay model | AoI-MDP · Underwater data collection task | AoI: 2.67±0.26 -> 1.97±0.26 | Helped |
| Use the SSP delay model instead of a geometric delay model | AoI-MDP · Underwater data collection task | AoI: 2.38±0.28 -> 1.97±0.26 | Helped |
| Use the SSP delay model instead of a Poisson delay model | AoI-MDP · Underwater data collection task | SDR: 5.95±2.42 -> 11.99±0.73 | Helped |
| Use the SSP delay model instead of an exponential delay model | AoI-MDP · Underwater data collection task | SDR: 7.65±1.99 -> 11.99±0.73 | Helped |
| Use a geometric delay model instead of the SSP delay model | AoI-MDP · Underwater data collection task | SDR: 11.99±0.73 -> 12.34±0.79 | Helped |
| Use the SSP delay model instead of a Poisson delay model | AoI-MDP · Underwater data collection task | EC: 34.27±7.98 -> 33.83±2.59 | Helped |
| Use the SSP delay model instead of an exponential delay model | AoI-MDP · Underwater data collection task | EC: 43.11±4.16 -> 33.83±2.59 | Helped |
Limitations
- Evaluated only on a simulated multi-AUV data collection task because open-source underwater tasks are scarce.
- Short student abstract; RL algorithm and training details are largely deferred to prior work.
Experimental setup
Datasets: multi-AUV data collection simulation environment (Zhang et al. 2024)
Benchmarks: multi-AUV underwater data collection simulation
Metrics: average AoI energy consumption sum data rate cumulative reward
Baselines: standard MDP SSP delay model Poisson delay model Exponential delay model Geometric delay model
Builds on: Markov decision process age of information
Closest papers by meaning
- Bounded Optimal Exploration in MDP (AAAI 2016 · similarity 0.85)
- Model-Based Exploration in Monitored Markov Decision Processes (ICML 2025 · similarity 0.85)
- Bayesian Policy Optimization for Model Uncertainty (ICLR 2019 · similarity 0.84)
- Sequential Monte Carlo for Policy Optimization in Continuous POMDPs (NeurIPS 2025 · similarity 0.83)
- UACOF: A USV-AUV Collaboration Framework for Underwater Tasks Under Extreme Sea Conditions (Student Abstract) (AAAI 2025 · similarity 0.83)
- Reducing AUV Energy Consumption Through Dynamic Sensor Directions Switching via Deep Reinforcement Learning (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.