Research map / Reinforcement learning / MDPs and planning

Actions Speak Louder Than Words: Rate-Reward Trade-off in Markov Decision Processes

ICLR 2025 · Reinforcement learning › MDPs and planning · approach: Average-reward and regularized MDPs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Characterizes the capacity-reward trade-off for communicating through MDP actions (action-state channel) and proposes Act2Comm, a transformer-based learned joint control/coding scheme that embeds messages into a target policy's actions.

Problem

Communicating messages implicitly through an agent's actions in an MDP, where the receiver observes only states, creating a trade-off between transmission rate/reliability and average reward.

Approach

Treats the MDP as a finite-state channel with actions as input and next states as output; derives a single-letter capacity and a concave rate-reward formulation, then learns a transformer encoder/decoder with a critic network that handles the non-differentiable channel and quantizer, while the encoder's policy mimics a target control policy.

Key results

• In Catch the Ball with p=0, Act2Comm communicates at rate 0.2 with zero error while reward drops from 1.66 to 1.5.
• Relaxing BER to 1e-4 achieves the same rate with reward 1.6 in Catch the Ball.
• Lowering the reward constraint yields more reliable communication at the same rate in Lucky Wheel.

Measured effects reported in the paper

Surprising findings

• The capacity of the action-state channel is achieved by a stationary randomized policy without relying on history.

Limitations

• Achieving capacity requires infinite code length, so the theory gives only an upper bound for practical codes.
• Communication through actions costs control performance and can drive the policy to deterministic behavior with zero communication capability when the reward constraint is tight.

Experimental setup

Benchmarks: Lucky Wheel Catch the Ball Erratic robot

Metrics: bit error rate average reward coding rate

Builds on: finite-state channel capacity theory Shannon's method for channel conversion emergent communication

Closest papers by meaning

• Communicating via Markov Decision Processes (ICML 2022 · similarity 0.90)
• Multi-Agent Reinforcement Learning with Communication-Constrained Priors (NeurIPS 2025 · similarity 0.88)
• Multi-Agent Coordination via Multi-Level Communication (NeurIPS 2024 · similarity 0.86)
• Narrowing the Collaboration Gap, Probably (NeurIPS 2026 · similarity 0.86)
• RGMComm: Return Gap Minimization via Discrete Communications in Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.85)
• Learning Weakly Communicating Average-Reward CMDPs: Strong Duality and Improved Regret (NeurIPS 2026 · similarity 0.85)