Research map / Large language models / Language, safety and interpretability

Emergent Representations of Program Semantics in Language Models Trained on Programs

ICML 2024 · Large language models › Language, safety and interpretability · approach: Reasoning mechanisms and failure modes · 9 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains a 350M CodeGen-style Transformer on synthetic Karel programs with input-output specifications and shows, via probes and a semantic-intervention control, that hidden states encode abstract program traces that emerge during training.

Problem

Whether language models of code trained only with next-token prediction on text learn the formal (operational) semantics of programs, or merely surface statistics, and how to control for what probes learn versus what the LM represents.

Approach

Probe LM hidden states for abstract program states (position, direction, obstacle) during training and introduce semantic probing interventions: reassign operation semantics (flip, adversarial) while keeping syntax fixed, and check whether a probe can still decode the alternative semantics as well as the original; a drop indicates the LM represents the true semantics rather than a syntactic record

Key results

• The LM reaches 92.4% generative accuracy at the end of training, with three phases (babbling, syntax acquisition, semantics acquisition).
• Semantic content of probes correlates strongly with generative accuracy in the second half of training (R^2 = 0.904, 0.886, 0.821 for linear, 1-layer MLP, 2-layer MLP).
• Under the semantic intervention, probes for flip and adversarial semantics achieve significantly lower semantic content than the original semantics in all 5 abstract-state cases at end of training, rejecting the syntactic record hypothesis.

Measured effects reported in the paper

Surprising findings

• Probes achieve above-baseline accuracy for predicting future abstract states, suggesting the LM encodes intended upcoming program states before generating them.
• Apparently high semantic content during the babbling phase disappears after the flip intervention, indicating it was attributable to the probe learning a few repeated programs.

Limitations

• Restricted to a simple straight-line Karel subset with no control flow and an 8x8 grid, so generalization to real languages is untested.
• Probe-based semantic content is an indirect measure and can still be influenced by probe capacity, with nonlinear probes requiring the intervention control.
• Results come from one model trained from scratch at one scale (350M parameters).

Experimental setup

Datasets: synthetic Karel program corpus (500,000 programs) test set of 10,000 specifications

Benchmarks: Karel synthetic test set (10,000 specifications)

Metrics: generative accuracy probing accuracy (semantic content, geometric mean over dire R^2 correlation with generative accuracy perplexity output diversity

Baselines: syntactic record hypothesis flip semantics adversarial semantics retrieval hypothesis

Builds on: CodeGen Karel Othello-GPT probing control tasks for probing

Closest papers by meaning

• ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding (ICLR 2025 · similarity 0.88)
• Language Models Can Predict Their Own Behavior (NeurIPS 2025 · similarity 0.87)
• Training Language Models to Explain Their Own Computations (NeurIPS 2026 · similarity 0.87)
• Do LLMs Build World Representations? Probing Through the Lens of State Abstraction (NeurIPS 2024 · similarity 0.87)
• Monitoring Latent World States in Language Models with Propositional Probes (ICLR 2025 · similarity 0.86)
• CODE REPRESENTATION LEARNING AT SCALE (ICLR 2024 · similarity 0.86)