Research map / Reinforcement learning / Visual and representation RL

Behavioral Foundation Models for Quality Diversity

NeurIPS 2026 · Reinforcement learning › Visual and representation RL · approach: Inferring decision-making from behavior · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BFM-QD, which runs quality-diversity search in the latent space of a frozen pretrained behavioral foundation model (FB or TD-JEPA) and adds a closed-form backward-inference variation operator that replaces critic-based policy gradients.

Problem

Quality-diversity methods search high-dimensional policy parameter spaces where random perturbations rarely yield rewarding behaviors, especially under sparse or deceptive rewards, so they fail on hard continuous-control tasks.

Approach

Relocate QD search from policy parameters to the compact latent space of a pretrained BFM, and use the closed-form least-squares task inference z* (backward inference) as a gradient-free directed variation operator that approximates a second-order policy improvement step.

Key results

• FB-ME and other BFM-QD variants reach the highest normalized QD-score on all tasks; on AntMaze-medium and Cube-single parameter-space baselines collapse to near-zero QD-score while BFM-QD maintains high performance.
• FB-BI (closed-form backward inference, no critic) matches FB-PGA-ME across all environments, while PGA-ME fails on AntMaze-medium and Cube-single.
• QD performance saturates within the first ~40k pretraining steps of FB, well before zero-shot performance converges.

Surprising findings

• Interpolation (isoline operator) helps only in the BFM latent space and yields no consistent gain in parameter space.
• Random exploration datasets outperform expert datasets for BFM pretraining.

Limitations

• Frozen BFM imposes a ceiling on behavioral expressivity; behaviors outside the pretrained latent space are inaccessible.
• Performance degrades with narrower datasets and with QD-collected data instead of RND, since latent diversity is bounded by data diversity.
• Each BFM is trained for a single environment, so environment-agnostic QD remains open; rewards and descriptors are assumed given.

Experimental setup

Datasets: OGBench antmaze-medium-explore OGBench cube-single-noisy RND-collected Walker2d and HalfCheetah data

Benchmarks: Walker2d_uni HalfCheetah_uni AntMaze-medium Cube-single

Metrics: QD-score coverage max fitness

Baselines: ME CMA-ME DCRL-ME DDE-Elites ME+Pretrain PGA-ME

Builds on: MAP-Elites CMA-ME PGA-ME Forward-Backward representations TD-JEPA DCRL-ME DDE-Elites

Closest papers by meaning

• Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization (ICML 2024 · similarity 0.88)
• Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control (NeurIPS 2024 · similarity 0.87)
• Representation-Based Exploration for Language Models: From Test-Time to Post-Training (ICLR 2026 · similarity 0.87)
• ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.87)
• Learning a Diffusion Model Policy from Rewards via Q-Score Matching (ICML 2024 · similarity 0.87)
• Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics (ICML 2024 · similarity 0.87)