Research map / Vision-language and multimodal / Human and embodied modeling
MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MM-OR, a large multimodal operating-room dataset with RGB-D, audio, speech, robot logs, tracking and panoptic/scene-graph annotations, and MM2SG, a multimodal LLM for scene graph generation.
Problem
Existing operating room datasets are small, lack realism, and do not capture the multimodal nature of OR scenes (audio, robot logs, tracking), limiting holistic OR scene graph modeling.
Approach
A large multimodal OR dataset with synchronized multi-view RGB-D, detail views, low-exposure camera, audio, speech transcripts, robot logs and infrared tracking, plus an MLLM (MM2SG) that encodes each modality into tokens and autoregressively generates scene graph triplets, trained with modality dropping and multimodality mixing augmentations.
Key results
- MM2SG with all modalities and temporality reaches macro F1 0.534 on MM-OR scene graph generation, versus 0.328 for PSG and 0.472 for ORacle.
- Modality dropping and multimodality mixing raise MM2SG F1 from 0.436 to 0.520.
- Segmentation VPQ reaches up to 71.8 when trained on both MM-OR and 4D-OR with offline temporal context (window 0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MM2SG instead of the PSG baseline | MM-OR scene graph generation · MM-OR test | macro F1: 0.328 -> 0.529 | Helped |
| Use MM2SG instead of ORacle | MM-OR scene graph generation · MM-OR test | macro F1: 0.472 -> 0.529 | Helped |
| Add point-cloud input to the camera inputs (ablation) | MM2SG · MM-OR test | macro F1: 0.484 -> 0.494 | Helped |
| Add audio and speech-transcript inputs (ablation) | MM2SG · MM-OR test | macro F1: 0.494 -> 0.484 | Hurt |
| Add robot-log and tracking inputs (ablation) | MM2SG · MM-OR test | macro F1: 0.484 -> 0.502 | Helped |
| Add predicted segmentation masks (ablation) | MM2SG · MM-OR test | macro F1: 0.502 -> 0.520 | Helped |
| Add temporal context to the all-modality model with predicted masks (ablation) | MM2SG · MM-OR test | macro F1: 0.520 -> 0.529 | Helped |
| Replace predicted masks with ground-truth masks (ablation) | MM2SG · MM-OR test | macro F1: 0.520 -> 0.519 | No clear effect |
| Add temporal context when using ground-truth segmentation (ablation) | MM2SG · MM-OR test | macro F1: 0.519 -> 0.534 | Helped |
| Add modality dropping during training (ablation) | MM2SG · MM-OR test | macro F1: 0.436 -> 0.469 | Helped |
| Add multimodality mixing to modality dropping (ablation) | MM2SG · MM-OR test | macro F1: 0.469 -> 0.520 | Helped |
| Use full temporal context instead of no temporal context (ablation) | DVIS++ trained on MM-OR · MM-OR video segmentation | VPQ (window 0/4/8): 64.6/59.6/59.3 -> 67.5/67.0/66.4 | Helped |
Surprising findings
- Training segmentation on combined MM-OR and 4D-OR improved performance over MM-OR alone, suggesting cross-dataset generalization.
Limitations
- Only total and partial knee replacements recorded in one OR, limiting generalization to other procedures and settings.
- Acquisition setup is costly and complex to scale to diverse or real-time hospital use.
- Rare relations perform poorly due to long-tailed predicate distribution.
Experimental setup
Datasets: MM-OR 4D-OR
Benchmarks: MM-OR 4D-OR
Metrics: macro F1 Video Panoptic Quality (VPQ) F1
Baselines: PSG ORacle DVIS++
Builds on: ORacle LLaVA PSG 4D-OR DVIS++
Closest papers by meaning
- EgoExOR: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understanding (NeurIPS 2025 · similarity 0.88)
- Bridging Vision and Language for Robust Context-Aware Surgical Point Tracking: The VL-SurgPT Dataset and Benchmark (AAAI 2026 · similarity 0.85)
- Multi-scale Temporal Prediction via Incremental Generation and Multi-agent Collaboration (NeurIPS 2025 · similarity 0.85)
- CholecTrack20: A Multi-Perspective Tracking Dataset for Surgical Tools (CVPR 2025 · similarity 0.85)
- SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark (CVPR 2026 · similarity 0.84)
- Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.