Research map / Reinforcement learning / Visual and representation RL
Inverse Reinforcement Learning with Switching Rewards and History Dependency for Characterizing Animal Behaviors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SWIRL, an EM-based inverse RL framework that models animal behavior as switching hidden-mode MDPs with state-dependent mode transitions and history-dependent (non-Markovian) rewards and policies.
Problem
Existing time-varying IRL methods for animal behavior ignore that animals' decisions depend on their history, and they cannot segment long, freely moving behavior into switching decision-making processes.
Approach
Extends multi-intention IRL on a hidden-mode MDP by making hidden-mode transitions depend on the current state (decision-level dependency) and making rewards and policies depend on the previous L states via state augmentation (action-level dependency), inferred with EM and soft-Q iteration under MaxEnt IRL.
Key results
- On the simulated gridworld, only the full history-dependent state-dependent model (S-2) accurately recovered reward maps, outperforming MaxEnt and the independent-transition variants on test LL and segmentation accuracy.
- On the mouse labyrinth data, SWIRL (S-2) recovered interpretable water, home, and explore reward maps with three hidden modes and outperformed MaxEnt and baselines on held-out test LL.
- On the spontaneous behavior dataset, SWIRL variants outperformed ARHMM and rARHMM in held-out test LL, but history-dependent variants (S-2) had lower LL than state-only variants (S-1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add state-dependent hidden-mode transitions and history-dependent rewards and policies (ablation) | SWIRL · 5×5 Gridworld | test log-likelihood | Helped |
| Add state-dependent hidden-mode transitions and history-dependent rewards and policies (ablation) | SWIRL · 5×5 Gridworld | test segmentation accuracy | Helped |
| Add history-dependent rewards and policies (L=2) (ablation) | SWIRL with state-dependent transitions (S-1) · 5×5 Gridworld | test log-likelihood | Helped |
| Add state-dependent hidden-mode transitions (ablation) | SWIRL with history-dependent rewards (I-2) · 5×5 Gridworld | test segmentation accuracy | Helped |
| Add state-dependent hidden-mode transitions and history-dependent rewards and policies (ablation) | SWIRL · Water-restricted labyrinth | held-out test log-likelihood | Helped |
| Add history-dependent rewards and policies (L=2) (ablation) | SWIRL with state-dependent transitions (S-1) · Water-restricted labyrinth | held-out test log-likelihood | Helped |
| Increase action-level history length from L=2 to L=4 (ablation) | SWIRL S-2 · Water-restricted labyrinth | held-out test log-likelihood | Helped |
| Increase action-level history length from L=1 to L=2 (ablation) | SWIRL S-1 · Mouse spontaneous behavior | held-out test log-likelihood | Hurt |
| Add state-dependent hidden-mode transitions (ablation) | SWIRL I-1 · Mouse spontaneous behavior | held-out test log-likelihood | Helped |
| Add history-dependent rewards and policies (ablation) | SWIRL with state-dependent transitions (S-1) · Mouse spontaneous behavior | held-out test log-likelihood | Hurt |
| Use discount factor γ=0.7 instead of γ=0.95 (ablation) | SWIRL · Mouse spontaneous behavior | held-out test log-likelihood | No clear effect |
| Use model-free IQ-Learn instead of model-based Soft-Q iteration (ablation) | SWIRL S-2 · 5×5 Gridworld | reward correlation: 0.737 | Mixed |
Surprising findings
- On the spontaneous behavior dataset, the history-dependent reward variants had lower test log-likelihood than the Markovian ones, which the authors attribute to preprocessing (merging identical syllables) making the time interval ill-defined.
- Model-free IQ-Learn-based SWIRL achieved a reward correlation of 0.737 on the gridworld, with noisier recovered rewards than the model-based version.
Limitations
- Not a formal POMDP method; history dependence is handled by state augmentation with a fixed history length L.
- Model-based soft-Q iteration is used in main experiments, which may not scale to large environments.
- History dependency did not improve test LL on the spontaneous behavior dataset, where the time concept after preprocessing is poorly defined.
Experimental setup
Datasets: simulated 5x5 gridworld water-restricted labyrinth mouse trajectories mouse spontaneous behavior MoSeq syllables with dopamine recordings
Benchmarks: simulated 5x5 gridworld water-restricted labyrinth mouse spontaneous behavior
Metrics: Pearson correlation of recovered reward test log-likelihood test segmentation accuracy
Baselines: MaxEnt IRL Multi-intention IQL Locally Consistent IRL ARHMM rARHMM
Builds on: maximum entropy inverse reinforcement learning multi-intention inverse reinforcement learning autoregressive hidden Markov model IQ-Learn
Closest papers by meaning
- Dynamic Inverse Reinforcement Learning for Characterizing Animal Behavior (NeurIPS 2022 · similarity 0.90)
- Intrinsic Goals for Autonomous Agents: Model-Based Exploration in Virtual Zebrafish Predicts Ethological Behavior and Whole-Brain Dynamics (NeurIPS 2025 · similarity 0.85)
- Flexible inference for animal learning rules using neural networks (NeurIPS 2025 · similarity 0.85)
- Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environments (NeurIPS 2025 · similarity 0.85)
- Hindsight PRIORs for Reward Learning from Human Preferences (ICLR 2024 · similarity 0.84)
- Abstract and Explore: A Novel Behavioral Metric with Cyclic Dynamics in Reinforcement Learning (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.