Research map / Reinforcement learning / Visual and representation RL

Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making

ICML 2025 · Reinforcement learning › Visual and representation RL · approach: In-context and adaptive RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ACE, a co-evolution framework where an LLM (as offline Policy Actor and Value Critic) refines RL trajectories and fine-tunes via LoRA on a mixed replay buffer, evaluated on three L2RPN power grid challenges.

Problem

Large-scale industrial decision making (power grid operation with >60K discrete actions) where LLMs lack real-time long-sequence control and RL suffers from poor sample efficiency in vast action spaces.

Approach

LLMs are used only offline during training: as a Policy Actor that rewrites suboptimal actions (validated by simulation) and as a Value Critic that reshapes trajectory-level rewards; refined transitions go into a mixed buffer with reward-based prioritization that both trains the SAC agent and LoRA-fine-tunes the LLM, while deployment remains LLM-free.

Key results

• On WCCI 2020, ACE (Qwen2-7B + SFT) reaches 69.8 episode reward and 92.9% survival versus 57.1 and 78.3% for expert-guided RL, and 64.4 for LLM4Teach.
• On WCCI 2022, ACE (Qwen2-7B + SFT) achieves 75.9 reward and 54.3% survival versus 31.0 for expert-guided RL, about 145% improvement.
• Pure LLM baselines (Qwen2-7B, GPT-4) reach only 21.9 to 29.5 reward on WCCI 2020, and ACE (GPT-4) keeps test time at 38.7s versus 1480 to 3415s for pure LLMs.

Measured effects reported in the paper

Surprising findings

• Pure LLM decision making is far worse than RL on full episodes, yet LLM refinements on only a few hundred selected transitions markedly speed up RL convergence.
• Action-based trajectory selection for the critic performed much worse than reward-based or state-based selection, suggesting LLMs struggle with abstract topology changes.

Limitations

• Pure LLM decision making fails over long horizons (800-2000 steps), so the LLM is restricted to offline refinement.
• Results depend on specific power grid environments (Grid2Op) and a single RL backbone; generalization to other industrial domains is not shown.
• Reported runs are with GPT-4o and Qwen2-7B; hyperparameter sensitivity (thresholds, intervals) is substantial.

Experimental setup

Datasets: L2RPN WCCI 2020 (Grid2Op) L2RPN NeurIPS 2020 (Grid2Op) L2RPN WCCI 2022 (Grid2Op)

Benchmarks: L2RPN WCCI 2020 L2RPN NeurIPS 2020 L2RPN WCCI 2022

Metrics: episode reward survival rate (%) sample requirements test time (s)

Baselines: Expert-guided RL (WCCI 2020 winning solution) Qwen2-7B-Instruct GPT-4o-0806 LLM4Teach

Builds on: Soft Actor-Critic LLM4Teach Grid2Op TD(lambda)

Closest papers by meaning

• On the Modeling Capabilities of Large Language Models for Sequential Decision Making (ICLR 2025 · similarity 0.90)
• From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.90)
• Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.89)
• Enhancing Decision-Making of Large Language Models via Actor-Critic (ICML 2025 · similarity 0.89)
• LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models (NeurIPS 2025 · similarity 0.89)
• Efficient Reinforcement Learning with Large Language Model Priors (ICLR 2025 · similarity 0.88)