Research map / Reinforcement learning / MDPs and planning

RL-CFR: Improving Action Abstraction for Imperfect Information Extensive-Form Games with Reinforcement Learning

ICML 2024 · Reinforcement learning › MDPs and planning · approach: POMDPs and confounded RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RL-CFR, which uses an actor-critic RL agent to dynamically choose action abstractions (public state as state, action-abstraction feature vector as action, payoff difference vs. default abstraction as reward) and then solves the resulting game tree with DCFR, evaluated on heads-up no-limit Texas Hold'em.

Problem

Fixed action abstractions in large imperfect-information extensive-form games (IIEFGs) are sub-optimal, while dynamic action abstraction methods suffer from poor CFR convergence and limited applicability.

Approach

Formulate action abstraction selection as an MDP whose states are public states and whose actions are continuous feature vectors mapped to raise sizes; the reward is the PBS value difference between the RL-selected abstraction and a default abstraction, computed by solving two ReBeL-style depth-limited subgames. The actor-critic network is trained from scratch with only game rules, and CFR is then

Key results

• RL-CFR beats ReBeL's replication by 64 ± 11 mbb/hand over 600,000 hands.
• RL-CFR beats Slumbot by 84 ± 17 mbb/hand over 250,000 hands, versus 18 ± 16 for the ReBeL replication.
• RL-CFR outperforms MUL-ACTION by 21 mbb/hand at about 1/3 of its running time, and FINE-GRAIN by 23 mbb/hand at about 2/3 of its running time.

Measured effects reported in the paper

Surprising findings

• The reported exploitability of RL-CFR (17 mbb/hand) is lower than that of the ReBeL replication (20 mbb/hand).

Limitations

• Evaluation is limited to heads-up no-limit Texas Hold'em, so generality to other IIEFGs is not shown.
• Results against Slumbot depend on an online opponent with nested subgame solving and high variance, and the ReBeL replication is weaker than the original ReBeL.
• Action network selection relies on an extra evaluation step and default-abstraction fallback, indicating the learned selection is not always reliable.

Experimental setup

Datasets: self-play HUNL PBS data generated by the method itself

Benchmarks: Heads-up No-limit Texas Hold'em head-to-head matches vs ReBeL replication and Slumbot

Metrics: win-rate (mbb/hand) exploitability (mbb/hand) running time

Baselines: ReBeL (replication) Slumbot ReBeL (original) MUL-ACTION FINE-GRAIN

Builds on: ReBeL DCFR counterfactual regret minimization actor-critic nested subgame solving

Closest papers by meaning

• Blending Imitation and Reinforcement Learning for Robust Policy Improvement (ICLR 2024 · similarity 0.87)
• Learning Imperfect Information Extensive-form Games with Last-iterate Convergence under Bandit Feedback (ICML 2025 · similarity 0.87)
• A Theoretical Framework for Partially-Observed Reward States in RLHF (ICLR 2025 · similarity 0.86)
• A Policy-Gradient Approach to Solving Imperfect-Information Games with Best-Iterate Convergence (ICLR 2025 · similarity 0.86)
• Action abstractions for amortized sampling (ICLR 2025 · similarity 0.86)
• Model-Based RL for Mean-Field Games is not Statistically Harder than Single-Agent RL (ICML 2024 · similarity 0.86)