Research map / Architectures and efficiency / RNNs and normalization
Scaling Effects on Latent Representation Edits in GPT Models (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a linear probe on a 12-layer GPT-2 trained on UCI-encoded chess moves and edits residual-stream activations along probe weight vectors, finding that intervention scale should decay exponentially with input length to keep moves legal.
Problem
How to edit a transformer's internal board representation via linear probes without making its outputs semantically invalid, specifically how intervention scale must vary with sequence position.
Approach
Uses linear probe weight columns as intervention vectors added to residual-stream activations to add or remove chess pieces, and empirically finds a position-dependent "Goldilocks" intervention scale well approximated by 0.3 + 0.7e^(-x) for move x.
Key results
- Probe exceeds 0.90 average across F1, accuracy, precision, and recall, and above 0.94 on layers 7 and 8.
- For moves after the first, a suitable intervention scale (e.g., 0.3) succeeds 92% of the time (sigma = 0.02).
- The center of the effective intervention scale region is approximated by H(x) = 0.3 + 0.7e^(-x).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale the probe-based intervention according to move position, using H(x) = 0.3 + 0.7e−x | Probe-based residual-stream edits in 12-layer GPT-2 · Chess, first 25 moves (50-ply) | legal move probability mass: 0.3 + 0.7e−x; 92% (σ = 0.02) | Helped |
| Use an intervention scale outside the narrow move-specific Goldilocks region | Probe-based residual-stream edits in 12-layer GPT-2 · Chess, across moves | legal move probability mass | Hurt |
| Use the same intervention scaling approach on the first move rather than a larger η | Probe-based residual-stream edits in 12-layer GPT-2 · Chess, first move | legal move probability mass | Hurt |
Surprising findings
- The first move requires a larger intervention scale to be effective, and over-scaling has few consequences for it.
Limitations
- Evaluated only on a chess domain with strict rules, not natural language.
- Intervention is highly sensitive to scale outside a narrow region that varies across moves.
- Small study (500 games, first 25 moves) with an empirically fitted scaling formula.
Experimental setup
Datasets: UCI-encoded chess move sequences human-played games hold-out set
Benchmarks: held-out human chess games (500 games sample)
Metrics: legal move probability mass (LMPM) probe F1, accuracy, precision, recall
Builds on: linear probes (Belinkov 2022) Othello-GPT probing (Nanda et al. 2023) chess GPT world models (Karvonen 2024)
Closest papers by meaning
- Deep Linear Probe Generators for Weight Space Learning (ICLR 2025 · similarity 0.83)
- On the Invariance and Generality of Neural Scaling Laws (NeurIPS 2026 · similarity 0.83)
- Understanding the Role of Training Data in Test-Time Scaling (ICLR 2026 · similarity 0.82)
- Probing Neural Combinatorial Optimization Models (NeurIPS 2025 · similarity 0.82)
- Neurons to Words: A Novel Method for Automated Neural Network Interpretability and Alignment (AAAI 2025 · similarity 0.81)
- Learning on Model Weights using Tree Experts (CVPR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.