Research map / Reinforcement learning / Policy gradient and actor-critic

A Model of Place Field Reorganization During Reward Maximization

ICML 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Stabilizing deep actor-critic · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a two-layer actor-critic model where Gaussian place fields are optimized online by the TD error, reproducing field density at rewards, backward field elongation, and drift, and showing noise in field updates helps new-target learning.

Problem

Explaining why hippocampal place fields densify at reward locations, elongate against the trajectory, and drift during stable behavior during navigation learning, which lacks a unifying normative account.

Approach

Place fields (amplitude, center, width) are Gaussian RBF features that feed a linear actor-critic and are updated online by the TD error jointly with actor and critic weights; small noise injected into field parameters both unifies the three phenomena and improves relearning of new targets.

Key results

• Optimizing place field widths and amplitudes improves cumulative reward (fixed-field agents plateau at G=33) and speeds policy convergence, while center optimization gives no benefit.
• Adding Gaussian noise (around 5e-4) to field parameter updates improves new-target learning monotonically up to a threshold of about 1e-3, beyond which reward decreases.
• Increasing noise magnitude causes population vector correlation to decrease rapidly while reward maximization behavior stays stable.

Measured effects reported in the paper

Surprising findings

• Optimizing place field centers did not improve and in fact decreased reward maximization and convergence speed when combined with amplitude optimization.
• Stable navigation behavior persists even as population vector correlation drops, indicating drift without performance cost.

Limitations

• Single-peaked Gaussian place fields rather than multi-field place cells.
• Field parameters are updated by backpropagating TD error, which is not biologically plausible as is.
• Tested only in a few simple 1D and 2D environments.

Experimental setup

Benchmarks: simulated 1D track navigation simulated 2D arena with obstacle

Metrics: cumulative discounted reward trials to policy convergence field density population vector correlation representation similarity correlation normalized field parameter variance

Baselines: Successor Representation agent Metric Representation agent fixed place field agent

Builds on: Foster et al. 2000 place field actor-critic model Wang et al. 2018 meta-RL joint objective successor representation Menache et al. 2005 basis function adaptation

Closest papers by meaning

• Time Makes Space: Emergence of Place Fields in Networks Encoding Temporally Continuous Sensory Experiences (NeurIPS 2024 · similarity 0.88)
• Global Distortions from Local Rewards: Neural Coding Strategies in Path-Integrating Neural Systems (NeurIPS 2024 · similarity 0.88)
• Trading Place for Space: Increasing Location Resolution Reduces Contextual Capacity in Hippocampal Codes (NeurIPS 2024 · similarity 0.87)
• A simple model of co-emergence of grid and place fields (NeurIPS 2026 · similarity 0.85)
• Place Cells as Multi-Scale Position Embeddings: Random Walk Transition Kernels for Path Planning (NeurIPS 2025 · similarity 0.85)
• REMI: Reconstructing Episodic Memory During Internally Driven Path Planning (NeurIPS 2025 · similarity 0.85)