Research map / Large language models / Language, safety and interpretability
Do LLMs Build World Representations? Probing Through the Lens of State Abstraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a state-abstraction-based probing framework and the REPLACE text planning task to test whether LLM representations keep full world state, world-irrelevant, Q*-irrelevant, or pi*-irrelevant abstractions, finding goal-oriented abstractions dominate after fine-tuning.
Problem
Prior probing studies of LLM world representations give conflicting conclusions because they probe for the complete world state without distinguishing which abstraction level (general vs. goal-oriented) the model actually encodes.
Approach
Applies RL state abstraction theory (world-irrelevant, Q*-irrelevant, pi*-irrelevant abstractions) to design probing targets, and uses a modular planning task so each abstraction level has a distinct, identifiable state space for probing LLM hidden states during decoding.
Key results
- SFT Llama2-13b reaches 88.30% success and 84.02% optimal rate on GRIPPER, while ICL reaches only 3.07% success.
- Probes recover Q*- and pi*-irrelevant predicates (e.g., nextObjDirect 92% recovery for Llama2-13b SFT) far better than raw/world-irrelevant predicates such as storeu (~8%).
- Advanced pre-training (Phi3-17b ICL) also favors goal-oriented abstractions, but raw store/held predicates stay near-identical across pre-trained models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Supervised fine-tune Llama2-13B instead of using in-context learning | Llama2-13B on GRIPPER · GRIPPER | nextObjDirect recovery rate: 17.51 -> 91.71 | Helped |
| Supervised fine-tune Llama2-13B instead of using in-context learning | Llama2-13B on COOK · COOK | nextObjDirect recovery rate: 9.59 -> 88.39 | Helped |
| Supervised fine-tune Llama2-13B instead of using in-context learning | Llama2-13B on GRIPPER · GRIPPER | storeu recovery rate: 3.52 -> 7.84 | No clear effect |
| Use advanced pre-training and larger model scale (Phi3-3.8B to Phi3-17B) | Phi3 on GRIPPER · GRIPPER | goal-oriented abstraction recovery rate | Helped |
| Make object color relevant to the goal with COLORGRIPPER (ablation) | Llama2-13B representations · GRIPPER | nearNotColor recovery rate: 0.00 -> 37.70 | Helped |
| Fine-tune on suboptimal action sequences containing random legal moves (ablation) | Llama3-8B on GRIPPER · GRIPPER | world-irrelevant recovery rate: 30.6 -> 27.4 | Hurt |
| Fine-tune on suboptimal action sequences containing random legal moves (ablation) | Llama3-8B on GRIPPER · GRIPPER | Q*-irrelevant recovery rate: 73.0 -> 63.3 | Hurt |
| Fine-tune on suboptimal action sequences containing random legal moves (ablation) | Llama3-8B on GRIPPER · GRIPPER | π*-irrelevant recovery rate: 87.8 -> 72.1 | Hurt |
| Train on COLORGRIPPER, where color is goal-relevant, rather than GRIPPER (ablation) | Llama2-13B representations · GRIPPER | color recovery rate: 0.00 -> 1.92 | No clear effect |
Surprising findings
- Probing store (world dynamics) remains poor even when fine-tuned models achieve near-perfect legal rates, implying models discard transition information.
- Fine-tuning on random suboptimal action sequences does not increase encoding of world-irrelevant abstraction.
Limitations
- Synthetic, relatively simple planning task unlike real NLP tasks
- Findings specific to the tested LLMs and task; probing cannot show the model actually uses the abstraction
- Only two datasets and a handful of open LLMs evaluated
Experimental setup
Datasets: GRIPPER (synthetic, REPLACE) COOK (synthetic, TextWorld-based)
Benchmarks: REPLACE GRIPPER REPLACE COOK
Metrics: legal rate success rate optimal rate F1-score recovery rate
Baselines: Transformer trained from scratch Pythia-70m Phi3-3.8b Phi3-17b
Builds on: state abstraction theory (RL) Li et al. 2021 probing world state in Othello transformers Li et al. 2023 discourse world state probing LoRA
Closest papers by meaning
- Can large language models explore in-context? (NeurIPS 2024 · similarity 0.90)
- Reinforcement World Model Learning for LLM-based Agents (NeurIPS 2026 · similarity 0.90)
- LLM-Empowered State Representation for Reinforcement Learning (ICML 2024 · similarity 0.89)
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.89)
- Emergent Response Planning in LLMs (ICML 2025 · similarity 0.89)
- From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.