Research map / Reinforcement learning / Visual and representation RL
Unlock the Cognitive Generalization of Deep Reinforcement Learning via Granular Ball Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GBRL, a framework that builds a VAE-based cognitive latent space from simple-scenario experience, segments it with granular ball representation, and approximates rewards/Q-values in complex scenarios to enable reward-agnostic fine-tuning of DRL policies.
Problem
Policies trained in simple scenarios often fail to transfer to complex scenarios with the same task logic; fine-tuning usually needs reward feedback that is costly to design in complex environments.
Approach
Build a dynamics-aware VAE latent space in the simple scenario, cluster it adaptively with granular ball representation, and in the complex scenario estimate rewards and Q-values of new samples from same-ball neighbors so TD-learning can proceed without reward feedback.
Key results
- GBRL-TD3 scored 6.17 on Ninja, 3.25 on Chaser, 5.63 on Heist, outperforming M2TD3, IDAAC and ExpGen on most tasks in Table 1 (3e5 fine-tuning steps, reward-given baselines).
- Using the proposed VAE with environmental dynamics loss outperformed vanilla VAE by over 20% on average across six ProcGen tasks (Table 2).
- GBR outperformed K-means, DBSCAN and DPC by about 15% on average across three ProcGen tasks (Table 9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GBRL-TD3 instead of ExpGen | ProcGen policies · Ninja hard | score: 6.13 ± 0.05 -> 6.17 ± 0.06 | Helped |
| Use GBRL-TD3 instead of ExpGen | ProcGen policies · Chaser hard | score: 3.48 ± 0.19 -> 3.25 ± 0.12 | Hurt |
| Use GBRL-TD3 instead of ExpGen | ProcGen policies · Heist hard | score: 5.47 ± 0.49 -> 5.63 ± 0.24 | Helped |
| Use GBRL-TD3 instead of IDAAC | ProcGen policies · Leaper hard | score: 5.32 ±1.04 -> 5.24 ± 1.07 | Hurt |
| Use GBRL-TD3 instead of ExpGen | ProcGen policies · Miner hard | score: 11.66 ± 2.74 -> 12.38 ± 1.47 | Helped |
| Use GBRL-TD3 instead of ExpGen | ProcGen policies · CaveFlyer hard | score: 6.16 ± 0.28 -> 6.13 ± 0.25 | Hurt |
| Use GBRL-TD3 instead of ExpGen | ProcGen policies · Dodgeball hard | score: 5.49 ± 0.25 -> 4.88 ± 0.13 | Hurt |
| Add environmental influence regularization to the VAE (ablation) | GBRL · ProcGen tasks (six reported) | score: 4.91 ± 0.18 -> 5.41 ± 0.53; 9.36 ± 2.85 -> 11.07 ± 2.97; 4.09 ± 0.72 -> 4.80 ± 0.41; 5.08 ± 0.31 -> 6.16 ± 0.11; 4.15 ± | Helped |
| Use cascaded instead of parallelized prediction (ablation) | VAE representation in GBRL · ProcGen tasks (Ninja, Chaser, Heist) | score: 5.85 ± 0.32 -> 6.11 ± 0.16; 2.93 ± 0.08 -> 3.09 ± 0.13; 4.15 ± 0.11 -> 5.27 ± 0.08 | Helped |
| Replace GBR clustering with K-means (ablation) | GBRL clustering · ProcGen tasks (Ninja, Chaser, Heist) | score: 5.22 ± 0.43 vs 6.16 ± 0.11; 2.86 ± 0.31 vs 3.27 ± 0.08; 2.71 ± 0.25 vs 3.27 ± 0.08 | Hurt |
| Replace GBR clustering with DBSCAN (ablation) | GBRL clustering · ProcGen tasks (Ninja, Chaser, Heist) | score: 5.79 ± 0.26 vs 6.16 ± 0.11; 2.53 ± 0.28 vs 3.27 ± 0.08; 3.04 ± 0.11 vs 3.27 ± 0.08 | Hurt |
| Replace GBR clustering with DPC (ablation) | GBRL clustering · ProcGen tasks (Ninja, Chaser, Heist) | score: 4.82 ± 0.51 vs 6.16 ± 0.11; 2.61 ± 0.41 vs 3.27 ± 0.08; 3.10 ± 0.14 vs 3.27 ± 0.08 | Hurt |
Surprising findings
- DDPG with GBRL sometimes exceeds the policy trained directly on the complex scenario, which the authors attribute to averaging neighbor Q-values mitigating Q overestimation.
Limitations
- GBR requires per-task tuning of minimum capacity eta and neighbor count k.
- Evaluated only on 2DMAZE and seven ProcGen tasks with a modified reward for baselines.
Experimental setup
Datasets: 2DMAZE (D4RL) ProcGen (Caveflyer, Chaser, Dodgeball, Heist, Leaper, Miner, Ninja)
Benchmarks: 2DMAZE (D4RL) ProcGen Ninja ProcGen Chaser ProcGen Heist ProcGen Leaper ProcGen Miner ProcGen CaveFlyer ProcGen Dodgeball
Metrics: episode score
Baselines: M2TD3 IDAAC ExpGen K-means DBSCAN DPC vanilla VAE GBRL
Builds on: DDPG TD3 PPO Granular Ball Representation VAE
Closest papers by meaning
- On Generalization Across Environments In Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.87)
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training (ICML 2025 · similarity 0.86)
- Zero Shot Generalization of Vision-Based RL Without Data Augmentation (ICML 2025 · similarity 0.86)
- Model-Based Transfer Learning for Contextual Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- Focus On What Matters: Separated Models For Visual-Based RL Generalization (NeurIPS 2024 · similarity 0.85)
- Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representations (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.