Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Ground-Compose-Reinforce, a neurosymbolic framework that grounds Reward Machine propositions from limited labelled trajectories, composes primitive value functions into optimal value estimates for new RM tasks, and uses them for self-generated shaped rewards in RL.
Problem
Training RL agents to perform novel, temporally extended behaviours specified in high-level language/formal task specs without manually designed reward functions, oracle labellers, or massive language-labelled datasets.
Approach
Ground atomic propositions via supervised classification on 350 labelled trajectories, learn primitive value functions for each proposition and its negation with offline RL, and compose them (max for disjunction, min for conjunction) to estimate optimal value functions for arbitrary Reward Machine tasks, enabling potential-based reward shaping that addresses propositional sparsity.
Key results
- GCR achieves 1.00 on GeoGrid Sequence, 5.36 on Loop (max expected 5.36), and 1538 on Hold-Red-Box and 1.00 on Pickup-Each-Box in DrawerWorld, while all non-compositional baselines score near zero on DrawerWorld.
- Bespoke Reward Model and Bespoke BC obtain 0 on all DrawerWorld tasks and 0 on Logic, showing failure with limited pretraining trajectories.
- Without reward shaping (No RS), GCR's Logic and Safety returns drop (0.00 and 0.23) and DrawerWorld returns are 0, showing shaping is critical for long-horizon propositional sparsity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Ground-Compose-Reinforce instead of non-compositional baselines | GeoGrid · GeoGrid Sequence, Loop, Logic, and Safety | undiscounted return: Sequence: 1.00 ± 0.00 vs 0.05±0.01; Loop: 5.36 ± 0.08 vs 0.04±0.01; Logic: 0.94±0.01 vs 0 ± 0; Safety: 1.00±0.00 vs −0.1 | Helped |
| Use Ground-Compose-Reinforce instead of non-compositional baselines | DrawerWorld · Hold-Red-Box, Pickup-Each-Box, and Show-Green-Box | undiscounted return: Hold-Red-Box: 1538 ± 130 vs 0 ± 0; Pickup-Each-Box: 1.00 ± 0.00 vs 0 ± 0; Show-Green-Box: 0.61 ± 0.06 vs 0 ± 0 | Helped |
| Add compositional reward shaping instead of using no reward shaping (ablation) | Ground-Compose-Reinforce · GeoGrid Sequence and Loop | undiscounted return: Sequence: 0.94 ± 0.03 -> 1.00 ± 0.00; Loop: 4.68±0.05 -> 5.36 ± 0.08 | Helped |
| Add compositional reward shaping instead of using no reward shaping (ablation) | Ground-Compose-Reinforce · GeoGrid Logic and Safety | undiscounted return: Logic: 0.00 ± 0.00 -> 0.94±0.01; Safety: 0.23±0.11 -> 1.00±0.00 | Helped |
| Add compositional reward shaping instead of using no reward shaping (ablation) | Ground-Compose-Reinforce · DrawerWorld Hold-Red-Box, Pickup-Each-Box, and Show-Green-Box | undiscounted return: Hold-Red-Box: 0 ± 0 -> 1538 ± 130; Pickup-Each-Box: 0 ± 0 -> 1.00 ± 0.00; Show-Green-Box: 0->0.61 ± 0.06 | Helped |
| Replace compositional reward shaping with RM-state-only High-Level RS (ablation) | Ground-Compose-Reinforce · GeoGrid Sequence, Loop, Logic, and Safety | undiscounted return: Sequence: 1.00 ± 0.00 vs 1.00 ± 0.00; Loop: 5.27 ± 0.08 vs 5.36 ± 0.08; Logic: 0.94 ± 0.01 vs 0.94±0.01; Safety: 0.97±0. | Mixed |
| Replace compositional reward shaping with RM-state-only High-Level RS (ablation) | Ground-Compose-Reinforce · DrawerWorld Hold-Red-Box and Pickup-Each-Box | undiscounted return: Hold-Red-Box: 0 ± 0 -> 1538 ± 130; Pickup-Each-Box: 0 ± 0 -> 1.00 ± 0.00 | Helped |
Surprising findings
- Bespoke Reward Model produces reward hacking in DrawerWorld, with high learned-reward returns but low ground-truth returns.
- GCR solves tasks (e.g., Pickup-Each-Box) that never appear in the pretraining data, where the best pretraining trajectory achieves zero return.
Limitations
- Assumes propositions can be faithfully grounded from the pretraining data; grounding errors propagate to reward generation.
- Propositions are fixed to a predefined set and labels are assumed available for a fixed vocabulary; representations like text propositions are future work.
- Reward hacking and ambiguity in propositions remain concerns; NL autoformalization only demonstrated zero-shot on GeoGrid.
Experimental setup
Datasets: GeoGrid pretraining dataset (5000 random-policy trajectories) DrawerWorld pretraining dataset (350 labelled trajectories)
Benchmarks: GeoGrid (Sequence, Loop, Logic, Safety) DrawerWorld (Hold-Red-Box, Pickup-Each-Box, Show-Green-Box)
Metrics: undiscounted return with respect to ground-truth rewards standard error over 5 runs
Baselines: LTL-conditioned Behaviour Cloning Bespoke Reward Model Bespoke Behaviour Cloning No RS High-Level RS
Builds on: Reward Machines PPO potential-based reward shaping fuzzy logic
Closest papers by meaning
- RLZero: Direct Policy Inference from Language Without In-Domain Supervision (NeurIPS 2025 · similarity 0.90)
- REvolve: Reward Evolution with Large Language Models using Human Feedback (ICLR 2025 · similarity 0.89)
- Language Grounded Multi-agent Reinforcement Learning with Human-interpretable Communication (NeurIPS 2024 · similarity 0.89)
- Representation-Based Exploration for Language Models: From Test-Time to Post-Training (ICLR 2026 · similarity 0.89)
- Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.89)
- Motif: Intrinsic Motivation from Artificial Intelligence Feedback (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.