Research map / Learning theory and algorithms / Submodular and game algorithms
Preference-CFR: Beyond Nash Equilibrium for Better Game Strategies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Pref-CFR, a CFR variant that uses per-action preference weights and a vulnerability (regret slack) parameter to steer imperfect-information game strategies toward chosen play styles while staying near Nash equilibrium.
Problem
CFR and its variants compute Nash equilibria that maximize payoff but ignore strategic diversity and user-preferred play styles; multiple equilibria or near-equilibria are not explored.
Approach
Introduce a per-information-set preference degree delta(I,a) that multiplies regrets to bias the strategy toward preferred actions, and a vulnerability degree beta that subtracts a slack from regrets so the final strategy is an epsilon-NE with epsilon <= beta, letting the user trade a bounded exploitability loss for a stylistic shift.
Key results
- In two-player Texas Hold'em, the loose AI raised fold-probability reduction from 5.4% to 0.3% preflop while head-to-head performance differences stayed within 10 mBB/h.
- In three-player Texas Hold'em the aggressive AI raised its 3-bet probability from 0.0% to 16.0% with about a 10 mBB/h decrement.
- In Kuhn poker, Pref-CFR converged to NE strategies different from CFR's alpha=0.2 when delta was set, with convergence speed comparable to CFR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Loose Passive Pref-CFR instead of standard CFR | Texas Hold'em · Two-player Texas Hold'em preflop | fold probability: 5.4% -> 0.3% | Helped |
| Use Aggressive Pref-CFR instead of standard CFR | Texas Hold'em · Two-player Texas Hold'em preflop | raise to 3 probability: 0.0% -> 21.1% | Helped |
| Use stylized Pref-CFR strategies instead of standard CFR | Texas Hold'em AIs · Two-player and three-player Texas Hold'em | head-to-head performance difference: within 10mBB/h | No clear effect |
| Use Loose Passive Pref-CFR instead of Standard CFR | Three-player Texas Hold'em · Three-player Texas Hold'em | earnings: 0.2mBB/h | Helped |
| Use Loose Passive Pref-CFR instead of Standard CFR | Three-player Texas Hold'em · Three-player Texas Hold'em | call probability: 0.0% -> 23.9% | Helped |
| Use Aggressive Pref-CFR instead of standard training | Three-player Texas Hold'em · Three-player Texas Hold'em | 3-bet probability: 0.0% -> 16.0% | Helped |
| Use Aggressive Pref-CFR instead of standard training | Three-player Texas Hold'em · Three-player Texas Hold'em | performance: 10mBB/h | Hurt |
| Set β>0 rather than β=0 (ablation) | Pref-CFR in Leduc poker · Leduc poker | strategy deviation from standard CFR | Helped |
| Set δ(raise)=10 rather than δ(raise)=5 (ablation) | Pref-CFR in Leduc poker · Leduc poker | raise probability | No clear effect |
| Use Pref-CFR(BR) rather than Pref-CFR(RM) (ablation) | Pref-CFR in Kuhn poker · Kuhn poker | performance | Helped |
| Use δ=10 rather than δ=5 (ablation) | Pref-CFR · Kuhn poker | deviation from the CFR equilibrium α=0.2 | Helped |
| Introduce β>0 alongside preference δ (ablation) | Pref-CFR · Leduc poker | strategy diversity | Helped |
Surprising findings
- Standard CFR converges to the same Kuhn poker equilibrium (alpha=0.2) regardless of initialization, and in Leduc poker preference alone (beta=0) did not change the converged strategy, only a positive vulnerability did.
- The loose-passive style gained 0.2 mBB/h over the normal strategy in three-player Texas Hold'em while raising calling from 0.0% to 23.9%.
Limitations
- Preference and vulnerability parameters must be manually tuned by experts per information set, and translating macro style into micro parameters is hard.
- Larger preference or vulnerability values slow convergence and increase exploitability against strong opponents.
- Convergence proofs rely on Blackwell approachability arguments and a conjecture; the Leduc result is limited to a single small game setting.
Experimental setup
Datasets: Kuhn poker Leduc poker Texas Hold'em (two-player and three-player)
Benchmarks: Kuhn poker Leduc poker two-player Texas Hold'em three-player Texas Hold'em
Metrics: exploitability mBB/h head-to-head win rate action frequencies
Baselines: CFR Kuhn poker NE strategies ES-MCCFR Pluribus
Builds on: counterfactual regret minimization CFR+ generalized weakened fictitious play Blackwell approachability
Closest papers by meaning
- Regret Minimization in Behaviorally-Constrained Zero-Sum Games (ICML 2017 · similarity 0.89)
- Strategy-Based Warm Starting for Regret Minimization in Games (AAAI 2016 · similarity 0.89)
- Accelerating Nash Equilibrium Convergence in Monte Carlo Settings Through Counterfactual Value Based Fictitious Play (NeurIPS 2024 · similarity 0.88)
- Learning Not to Regret (AAAI 2024 · similarity 0.87)
- A Minimaximalist Approach to Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.86)
- Sequential Decision Making in Stochastic Games with Incomplete Preferences over Temporal Objectives (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.