Research map / Multi-agent systems and games / Human-AI and goal-conditioned
BricksRL: A Platform for Democratizing Robotics and Reinforcement Learning Research and Education with LEGO
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents BricksRL, an open-source platform that connects LEGO hubs (via Pybricks over Bluetooth) to TorchRL so researchers and students can train custom LEGO robots end-to-end with online and offline RL in the real world.
Problem
Robotics and RL research and education are limited by the cost, setup complexity and steep learning curve of real robot labs; existing low-cost robot setups are static or require heavy tooling.
Approach
Interface LEGO hubs running Pybricks with TorchRL's EnvBase, agents and TensorDict through a BLE-based gym-like environment, so any LEGO build can be trained with off-policy RL (TD3, SAC, DroQ) on a laptop/GPU, with sim counterparts and offline datasets.
Key results
- On real LEGO robots, TD3, SAC and DroQ learned tasks such as RunAway-v0 and Spinning-v0 in roughly 10-15 minutes of training.
- RoboArm-v0 real-robot success rates were 88% (TD3), 72% (SAC) and 64% (DroQ) versus 32% for random, with simulation-trained policies reaching 100% (SAC*).
- In offline RL, expert-data IQL and CQL outperformed online algorithms on Walker-v0 and RoboArm-v0 per the authors' evaluation table, while online TD3/SAC struggled.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a trained online RL policy rather than a random policy | 2Wheeler agents · RunAway-v0 | evaluation reward: -0.51 ± 1.84; TD3 7.64 ± 2.31, SAC 8.72 ± 2.82, DroQ 8.96 ± 0.87 | Helped |
| Use a trained online RL policy rather than a random policy | 2Wheeler agents · Spinning-v0 | evaluation reward: 71.97 ± 501.79; TD3 7558.21 ± 28.61, SAC 7407.20 ± 109.53, DroQ 7456.85 ± 18.02 | Helped |
| Use a trained online RL policy rather than a random policy | Walker agents · Walker-v0 | evaluation reward: -191.99 ± 18.19; TD3 -62.94 ± 16.76, SAC -52.04 ± 8.79, DroQ -57.63 ± 10.44 | Helped |
| Train the policy in simulation instead of on the real robot | Walker agents · WalkerSim-v0, evaluated on real Walker | evaluation reward: TD3: -62.94 ± 16.76 -> -78.49 ± 9.75; SAC: -52.04 ± 8.79 -> -55.03 ± 4.95; DroQ: -57.63 ± 10.44 -> -56.62 ± 2.81 | Mixed |
| Train the policy in simulation instead of on the real robot | RoboArm agents · RoboArm-v0, evaluated on real robot | success rate (%): TD3: 88 -> 88; SAC: 72 -> 100; DroQ: 64 -> 88 | Mixed |
| Add webcam image observations to the RoboArm task | RoboArm agents · RoboArm-mixed-v0 | success rate (%): TD3: 88 -> 8; SAC: 72 -> 68; DroQ: 64 -> 68 | Mixed |
| Use a lower communication frequency (2 Hz) instead of 11 Hz | DroQ on Walker · Walker-v0 | convergence speed and stability | Helped |
| Use expert-data offline RL rather than online RL | Offline RL algorithms · Walker-v0 | evaluation reward: TD3: -79.71; SAC: -66.91; BC: -85.65; IQL: -74.80; CQL: -77.93 | Mixed |
| Use expert data rather than random data | Offline RL algorithms · RoboArm-v0 | evaluation reward: BC: -117.72 -> -7.34; IQL: -76.89 -> -3.10; CQL: -46.91 -> -17.41; TD3: -124.22 -> -201.94; SAC: -54.67 -> -218.76 | Mixed |
| Use expert data rather than random data | Offline RL algorithms · RunAway-v0 | evaluation reward: TD3: 19.86 -> 9.06; SAC: 14.74 -> 10.80; BC: -0.27 -> 18.13; IQL: 14.07 -> 18.80; CQL: 19.60 -> 19.74 | Mixed |
| Use expert data rather than random data | Offline RL algorithms · Spinning-v0 | evaluation reward: TD3: 6160.25 -> 6168.28; SAC: 5416.11 -> 9349.52; BC: 35.55 -> 9150.02; IQL: 4544.28 -> 9096.60; CQL: 4509.31 -> 9099.03 | Helped |
Surprising findings
- Training at a lower communication frequency (2 Hz) converged faster and more stably than at 11 Hz for the Walker task, which the authors attribute to a frame-skip-like effect.
- Online algorithms (TD3, SAC) performed worse than offline methods trained on expert data, which the authors suggest reflects overfitting or poor generalization.
Limitations
- Communication overhead via stdin/stdout reduces system frequency to about 11 Hz, far below the hub's 1000 Hz control loop.
- No hyperparameter optimization was performed; default settings were used across algorithms.
- Results are from few tasks and three robots with 5 seeds, and online algorithms showed high variance (e.g., TD3 failing on RoboArm-mixed-v0).
Experimental setup
Datasets: BricksRL offline datasets (expert and random) for Walker-v0, RoboArm-v0, RunAway-v0, Spinning-v0
Benchmarks: RunAway-v0 Spinning-v0 Walker-v0 WalkerSim-v0 RoboArm-v0 RoboArmSim-v0 RoboArm-mixed-v0
Metrics: mean episode evaluation reward success rate training time
Baselines: TD3 SAC DroQ Random policy BC IQL CQL
Builds on: TorchRL Pybricks TD3 SAC DroQ
Closest papers by meaning
- Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (AAAI 2025 · similarity 0.86)
- Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks (ICLR 2025 · similarity 0.85)
- TorchRL: A data-driven decision-making library for PyTorch (ICLR 2024 · similarity 0.84)
- Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning (ICML 2024 · similarity 0.84)
- Diffusion-based Curriculum Reinforcement Learning (NeurIPS 2024 · similarity 0.84)
- Text-Aware Diffusion for Policy Learning (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.