Research map / Large language models / AI and society
A Retrospective on the Robot Air Hockey Challenge: Benchmarking Robust, Reliable, and Safe Learning Techniques for Real-world Robotics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Retrospective on the NeurIPS 2023 Robot Air Hockey Challenge, a benchmark for safe, robust, real-time robot learning under sim-to-real factors, with results from qualifying and tournament stages and real-robot deployment of top teams.
Problem
Learning-based control for real robots faces sim-to-real gaps, safety constraints, limited data, real-time requirements and noise, which standard ML benchmarks often omit; the paper evaluates how competitors handle these in dynamic robot air hockey.
Approach
A competition-based benchmark on dynamic robot air hockey that penalizes safety/constraint violations via a deployability score, evaluates in a modified simulator with real-world factors (observation loss, tracking loss, model mismatch, disturbances), and validates top agents on a real Kuka robot setup.
Key results
- AiRLIHockey, a model-predictive-control-based solution, won the qualifying stage with 73.8 score and 54.9% Hit, 84.5% Defend, 90.3% Prepare success rates.
- In the tournament, AiRLIHockey won all 11 matches (270 goals scored, 8 received), while SpaceR (DreamerV3) placed second.
- Top three teams were successfully deployed on a real Kuka robot setup, with real-world performance considerably worse than simulation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine learning with robotics and physics-based prior knowledge rather than using a purely data-driven approach | Robot Air Hockey Challenge agents · Qualifying stage | Score | Helped |
| Use model predictive control and classical robotics techniques rather than the other methodologies | AiRLIHockey · Tournament stage | Total points: 33 vs 21 | Helped |
| Add puck tracking loss (ablation) | Submitted solutions · Qualifying tasks | Success rate | Hurt |
| Add disturbances to puck dynamics (ablation) | SpaceR and RL3_polimi · Qualifying tasks | Success rate | Helped |
| Add model mismatch on the robot arm (ablation) | Submitted solutions · Qualifying tasks | Success rate | No clear effect |
| Add observation noise (ablation) | Submitted solutions · Qualifying tasks | Success rate | No clear effect |
| Add all environment mismatch factors together (ablation) | Submitted solutions · Qualifying tasks | Success rate | Hurt |
| Add a single less relevant mismatch factor (ablation) | AiRLIHockey and Air-HocKIT · Qualifying tasks | Success rate | No clear effect |
| Replace a flat policy with a state-machine high-level policy | Air Hockey agents · Dynamic air hockey tasks | Performance | Helped |
Surprising findings
- Penalty-based safety specifications were brittle out-of-distribution, with RL3_polimi achieving the third-highest score but ranking low due to safety violations.
- Model mismatch played a marginal role in the sim-to-real gap, while tracking loss and disturbances mattered most.
Limitations
- Puck perception assumed perfect via OptiTrack motion capture, neglecting perception and action-perception coupling.
- Limited sample size of teams and analyses; few strong baselines.
- Real-world stage not part of competition and required hand tuning and engineering help, limiting real-world evaluation.
Experimental setup
Datasets: simulated air hockey rollouts from modified MuJoCo environment
Benchmarks: Robot Air Hockey Challenge qualifying tasks (Hit, Defend, Prepare) Robot Air Hockey Challenge tournament real-robot air hockey deployment
Metrics: success rate deployability score (penalty points) game score wins/goals
Baselines: Baseline (planning/optimization robotic baseline) Baseline-ATACOM SpaceR Air-HocKIT RL3_polimi
Builds on: ATACOM DreamerV3 PPO SAC PGPE
Closest papers by meaning
- Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (AAAI 2025 · similarity 0.86)
- RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Models (NeurIPS 2026 · similarity 0.84)
- AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation (ICLR 2025 · similarity 0.83)
- What Matters in Learning from Large-Scale Datasets for Robot Manipulation (ICLR 2025 · similarity 0.83)
- Efficient Robotic Policy Learning via Latent Space Backward Planning (ICML 2025 · similarity 0.83)
- RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.