Research map / Reinforcement learning / Visual and representation RL
SonoGym: High Performance Simulation for Challenging Surgical Tasks with Robotic Ultrasound
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SonoGym, a GPU-parallel robotic ultrasound simulator built from TotalSegmentator CT models with model-based and pix2pix-based ultrasound rendering, providing navigation, bone reconstruction, and drilling tasks for PPO, A2C, submodular PPO, safe RL, ACT, and diffusion policy.
Problem
Robot learning for complex robotic ultrasound tasks (anatomy reconstruction and ultrasound-guided orthopedic surgery) is limited by the lack of realistic, efficient simulation environments with ultrasound imaging tied to patient anatomy.
Approach
A scalable parallel simulator that renders ultrasound images from CT-derived 3D models both by physics-based ray-tracing and by a pix2pix GAN trained on paired CT-US data, and packages MDP formulations (POMDP, submodular MDP, state-wise constrained MDP) and expert datasets for RL and IL benchmarking.
Key results
- Learning-based US simulator achieves LPIPS 0.2415, SSIM 0.3940, PSNR 15.96; 100 parallel environments render in 0.0089 s (MB) and 0.1107 s (LB) on an RTX 3090 Ti.
- In surgery, PPO + safety filter achieves 2.17 mm side error and 100% safe ratio in-domain (MB), while ACT achieves the lowest insertion error (2.92 mm) and rotation error (0.62 deg).
- For reconstruction, learned submodular PPO trajectories achieve higher coverage with lower rotation angles and path length than the heuristic trajectory; generalization to an unseen patient in surgery degrades substantially.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a safety filter to PPO (ablation) | PPO on surgery with model-based ultrasound · Surgery MB IDT | safe ratio (%): 99.9 -> 100.0 | Helped |
| add a safety filter to PPO (ablation) | PPO on surgery with model-based ultrasound · Surgery MB IDT | insertion error (mm): 16.0 -> 13.9 | Helped |
| add a safety filter to PPO (ablation) | PPO on surgery with model-based ultrasound · Surgery MB ODT | side error (mm): 5.66 -> 8.94 | Hurt |
| add a safety filter to PPO (ablation) | PPO on surgery with learning-based ultrasound · Surgery LB IDT | safe ratio (%): 93.1 -> 95.3 | Helped |
| add a safety filter to PPO (ablation) | PPO on surgery with learning-based ultrasound · Surgery LB IDT | insertion error (mm): 12.3 -> 11.9 | Helped |
| use ACT instead of PPO | surgery with learning-based ultrasound · Surgery LB IDT | insertion error (mm): 12.3 -> 1.62 | Helped |
| use ACT instead of PPO | surgery with model-based ultrasound · Surgery MB IDT | safe ratio (%): 99.9 -> 65.9 | Hurt |
| train and test on a held-out patient rather than the same patient | surgery PPO with learning-based ultrasound · New patient | side error (mm): 5.42 -> 27.3 | Hurt |
| train and test on a held-out patient rather than the same patient | surgery PPO with learning-based ultrasound · New patient | safe ratio (%): 93.1 -> 52.86 | Hurt |
| use multiple ultrasound simulation networks during training | navigation agents · Navigation ODT | position error: less than $16[\text{mm}]$ | No clear effect |
| use SAC instead of PPO/A2C | navigation and surgery agents · Navigation and surgery | task performance | Hurt |
| vary Decision Transformer history, action steps, planning horizon, and learning rate | Decision Transformer on surgery · Surgery | task performance | No clear effect |
Surprising findings
- Generalization to an unseen patient is much worse than generalization to unseen ultrasound simulation networks, with large gaps in safe ratio and side error.
- SAC and PPO-Lagrangian failed to produce consistently successful policies on the navigation and surgery tasks despite extensive hyperparameter search.
Limitations
- GAN-based ultrasound lacks physics-based frame-to-frame consistency.
- Patient models are static and do not model soft tissue deformation.
- Results come from a limited number of patient models (10 subjects) and have not been validated on real robotic ultrasound systems.
Experimental setup
Datasets: TotalSegmentator in-house paired CT-ultrasound dataset from 7 ex-vivo spine specimens
Benchmarks: SonoGym navigation SonoGym bone surface reconstruction SonoGym ultrasound-guided surgery
Metrics: LPIPS SSIM PSNR final position error rotation error coverage ratio trajectory length insertion error side error safe ratio
Baselines: PPO A2C PPO + safety filter ACT Diffusion Policy heuristic open-loop trajectory SAC PPO-Lagrangian
Builds on: PPO A2C SafeRPlan Action Chunking Transformer Diffusion Policy pix2pix IsaacLab
Closest papers by meaning
- SurgicAI: A Hierarchical Platform for Fine-Grained Surgical Policy Learning and Benchmarking (NeurIPS 2024 · similarity 0.85)
- SurgVista: Long-Horizon Surgical World Modeling with Plausible Instrument-Tissue Dynamics (NeurIPS 2026 · similarity 0.84)
- EchoWorld: Learning Motion-Aware World Models for Echocardiography Probe Guidance (CVPR 2025 · similarity 0.83)
- U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding (ICLR 2026 · similarity 0.83)
- SutureBot: A Precision Framework & Benchmark For Autonomous End-to-End Suturing (NeurIPS 2025 · similarity 0.82)
- Intraoperative 2D/3D Image Registration via Differentiable X-ray Rendering (CVPR 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.