Research map / Video / Video-language understanding
Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CoopSR benchmark and EgoTeam dataset (114K QAs) for multi-robot cooperative egocentric spatial reasoning, and SP-CoR, an MLLM framework with spectral frame sampling, physics-informed multi-robot fusion, and pose-privileged prompt distillation.
Problem
MLLMs cannot reason cooperatively over synchronized egocentric video streams from multiple moving robots; concatenating streams is costly and lacks mechanisms for cross-robot alignment and fusion.
Approach
A training-free two-stage sampler (CLIP relevance then temporal FFT energy over CLIP features), a fusion block injecting FFT and pose-motion tokens, and privileged-information prompt distillation from a pose-aware teacher so that only egocentric video is needed at test time.
Key results
- SP-CoR (Qwen2.5-VL-7B) reaches 70.55% average accuracy on Habitat, +3.87% over the strongest baseline.
- SP-CoR reaches 70.82% on iGibson, +7.12% over the strongest baseline.
- Removing SPI-MRF drops iGibson accuracy from 70.82% to 55.12%.
Surprising findings
- A text-only baseline reaches only 29.47% on Habitat, showing the benchmark is not solvable from language priors alone.
- Dropping one robot at inference drops SP-CoR average accuracy from 70.55% to 53.29% on Habitat.
Limitations
- Real-world test set is limited in scene diversity, robot types and team sizes.
- Benchmark is primarily simulation-based.
- Dual-use and privacy concerns for multi-robot perception.
Experimental setup
Datasets: EgoTeam Habitat iGibson
Benchmarks: CoopSR (Habitat) CoopSR (iGibson) CoopSR real-world test
Metrics: MC4 accuracy per-tier accuracy per-team-size accuracy
Baselines: Qwen2.5-VL-7B-Instruct Qwen2.5-VL-72B-Instruct InternVL3-8B Qwen3-VL-8B-Instruct VideoLLaMA3-7B VideoLLaMA3-2B LLaVA-NeXT-Video-7B VideoRAG EgoRAG TSPO KTV GPT-5 mini
Builds on: LoRA CLIP Qwen2.5-VL VideoRAG EgoRAG TSPO
Closest papers by meaning
- Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.90)
- EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT (NeurIPS 2025 · similarity 0.90)
- RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics (CVPR 2025 · similarity 0.90)
- EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy (CVPR 2026 · similarity 0.90)
- Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes (ICLR 2026 · similarity 0.89)
- Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.