Research map / Reinforcement learning / Visual and representation RL
Flexible inference for animal learning rules using neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Infers animal learning rules from de novo mouse behavior by parameterizing the weight update of a Bernoulli GLM policy with a feedforward DNN (DNNGLM) or GRU-based RNN (RNNGLM), outperforming REINFORCE-based models on IBL held-out log-likelihood.
Problem
Existing approaches to inferring animal learning rules assume fixed parametric forms (e.g., Q-learning, REINFORCE) and are mostly limited to bandit tasks, so they cannot capture flexible or non-Markovian learning during de novo task acquisition.
Approach
Model the policy as a dynamic Bernoulli GLM whose per-trial weight update is a neural network function of current weights, covariates, choice and reward (or a GRU hidden state for non-Markovian dependence), fit end-to-end by maximizing choice log-likelihood.
Key results
- On IBL held-out animals, RNNGLM achieved total held-out LL of -59892 versus -60576 for REINFORCE and -60360 for DNNGLM (p=1.79e-8 DNNGLM vs RNNGLM).
- In simulation with a non-Markovian REINFORCE-variant ground truth, RNNGLM recovered reward-history dependence that DNNGLM could not capture.
- Reconstruction RMSE of the update function decreased as the number of simulated animals in the pool grew (Fig. C), with a noise-degradation table showing RMSE 0.0216 to 0.0652 across noise levels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace REINFORCE with DNNGLM | IBL mouse learning prediction · IBL mice, animal-held-out | held-out log-likelihood: -60576 -> -60360 | Helped |
| replace DNNGLM with RNNGLM to model trial history recurrently | IBL mouse learning prediction · IBL mice, animal-held-out | held-out log-likelihood: -60360 -> -59892 | Helped |
| constrain the baseline to be nonnegative (ablation) | REINFORCE on IBL mice · IBL mice, animal-held-out | held-out log-likelihood: -60576 -> -68532 | Hurt |
| use RNNGLM instead of tinyRNN | IBL mouse learning prediction · IBL mice, animal-held-out | held-out log-likelihood: -60564 -> -59892 | Helped |
| add previous-trial information to DNNGLM (ablation) | DNNGLM · IBL mice, animal-held-out | held-out log-likelihood: -60168 -> -59892 | Hurt |
| replace REINFORCE with DNNGLM | IBL mouse learning prediction · IBL mice, future-data holdout | held-out log-likelihood: -2924.4 -> -2905.8 | Helped |
| replace DNNGLM with RNNGLM to model trial history recurrently | IBL mouse learning prediction · IBL mice, future-data holdout | held-out log-likelihood: -2905.8 -> -2881.2 | Helped |
| constrain the baseline to be nonnegative (ablation) | REINFORCE on IBL mice · IBL mice, future-data holdout | held-out log-likelihood: -2924.4 -> -3272.4 | Hurt |
| add previous-trial information to DNNGLM (ablation) | DNNGLM · IBL mice, future-data holdout | held-out log-likelihood: -2890.2 -> -2881.2 | Hurt |
| replace tinyRNN with RNNGLM | IBL mouse learning prediction · IBL mice, future-data holdout | held-out log-likelihood: -2893.8 -> -2881.2 | Helped |
| use a Transformer instead of an RNN (ablation) | learning-rule inference architecture · IBL mice | performance: p=7.44e-1 | No clear effect |
| condition updates on reward history using a recurrent learning rule | RNNGLM · simulated non-Markovian learning rule | weight-update magnitude | Helped |
Surprising findings
- Enforcing a nonnegative baseline on the inferred update reduced held-out LL substantially (REINFORCE vs. nonnegative base: -60576 vs -68532), indicating the negative baseline is functionally meaningful.
- Tiny RNN (8 GRUs) baseline performed worse than RNNGLM and even worse in the authors' tests than in the original paper's 1-4 GRU setting.
Limitations
- Pooling across animals assumes a shared learning rule and may obscure individual variability; low-data regimes make inference underconstrained.
- The decision model is fixed to a GLM and the learning rule is assumed static and deterministic.
- Evaluation limited to a single simplified IBL dataset of 12 animals.
Experimental setup
Datasets: International Brain Laboratory (IBL) mouse sensory decision-making dataset simulated REINFORCE-generated choice data
Benchmarks: IBL mouse learning data (held-out animals) IBL future-data holdout
Metrics: held-out log-likelihood RMSE of reconstructed weight update paired t-test p-values
Baselines: REINFORCE DNNGLM (history) REINFORCE (history) tinyRNN PsyTrack
Builds on: REINFORCE PsyTrack tinyRNN GRU
Closest papers by meaning
- Model Based Inference of Synaptic Plasticity Rules (NeurIPS 2024 · similarity 0.87)
- Inverse Reinforcement Learning with Switching Rewards and History Dependency for Characterizing Animal Behaviors (ICML 2025 · similarity 0.85)
- Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learning (ICML 2024 · similarity 0.85)
- Deep Reinforcement Learning with Time-Scale Invariant Memory (AAAI 2025 · similarity 0.85)
- Rapid Learning without Catastrophic Forgetting in the Morris Water Maze (ICML 2024 · similarity 0.85)
- Dynamics of Supervised and Reinforcement Learning in the Non-Linear Perceptron (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.