Research map / Reinforcement learning / Visual and representation RL
LLM-Empowered State Representation for Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LESR, which uses GPT-4 to generate task-related state-representation and intrinsic-reward code for TD3 with LLM-feedback iterations guided by Lipschitz constants, evaluated on Mujoco and Gym-Robotics.
Problem
Standard state representations omit task-related details, so value networks map states to rewards poorly (low smoothness), hurting sample efficiency in RL.
Approach
An LLM writes python functions that augment the source state with task-related dimensions and an intrinsic reward over them; the LLM is iteratively refined using per-dimension Lipschitz constants of the state-to-reward mapping as feedback. The LLM is not queried during training or testing after the search stage.
Key results
- LESR averages 29% relative improvement in accumulated reward over TD3 on 5 Mujoco tasks at 1M steps (vs 7 per-task improvements averaging around -2% to 10% for EUREKA and ablations).
- LESR averages 30% success-rate improvement across 7 Gym-Robotics tasks, with TD3 getting 0 success on AntMaze tasks.
- On Swimmer, the LLM-generated state representation reduces the estimated Lipschitz constant from 560.2 to 168.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LESR-generated task-related state representations and intrinsic rewards | TD3 · Mujoco (mean, 1M steps) | RPI: 29% vs TD3; 29% vs -2% | Helped |
| Add LESR-generated task-related state representations and intrinsic rewards | TD3 · Gym-Robotics (mean, 1M steps) | PI: 30% vs TD3; 30% vs 8% | Helped |
| Add LESR state representation (ablation) | LESR · Mujoco (mean, 1M steps) | RPI: 21% -> 29% | Helped |
| Add LESR state representation (ablation) | LESR · Gym-Robotics (mean, 1M steps) | PI: 6% -> 30% | Helped |
| Add intrinsic reward (ablation) | LESR · Mujoco (mean, 1M steps) | RPI: 21% -> 29% | Helped |
| Add intrinsic reward (ablation) | LESR · Gym-Robotics (mean, 1M steps) | PI: 6% -> 30% | Helped |
| Add iterative feedback (ablation) | LESR · Mujoco (mean, 1M steps) | RPI: 3% -> 29% | Helped |
| Add iterative feedback (ablation) | LESR · Gym-Robotics (mean, 1M steps) | PI: 10% -> 30% | Helped |
| Increase sample count K from 1 to 3 (ablation) | LESR · AdroitHandDoor | performance | Helped |
| Increase sample count K from 3 to 9 (ablation) | LESR · AdroitHandDoor | performance | No clear effect |
| Vary intrinsic reward weight w (ablation) | LESR · AdroitHandDoor | performance | No clear effect |
| Use LESR state representation instead of source state | TD3 Ant critic/reward mapping · Ant | Lipschitz constant: 560.2 -> 168.1 | Helped |
Surprising findings
- Removing the intrinsic reward (w/o IR) has minimal effect on Mujoco performance, suggesting the state representation drives most gains.
- EUREKA underperforms TD3 on Walker2d and Ant in the 1M-step table (-9% and -27%).
Limitations
- Relies solely on source state features; cannot add external information for partially observable environments.
- Quality of generated representations bounded by the LLM's capability; no guarantee of task-relevant output.
- Evaluated only on Mujoco and Gym-Robotics with a single base algorithm (TD3) during search.
Experimental setup
Datasets: Mujoco Gym-Robotics
Benchmarks: Mujoco Gym-Robotics
Metrics: accumulated reward success rate relative performance improvement Lipschitz constant
Baselines: TD3 EUREKA PPO SAC
Builds on: TD3 EUREKA chain-of-thought prompting
Closest papers by meaning
- Do LLMs Build World Representations? Probing Through the Lens of State Abstraction (NeurIPS 2024 · similarity 0.89)
- Efficient Reinforcement Learning with Large Language Model Priors (ICLR 2025 · similarity 0.89)
- Reinforcement World Model Learning for LLM-based Agents (NeurIPS 2026 · similarity 0.89)
- From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.89)
- On the Modeling Capabilities of Large Language Models for Sequential Decision Making (ICLR 2025 · similarity 0.88)
- Learning Goal-Conditioned Representations for Language Reward Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.