Research map / Vision-language and multimodal / Cross-modal learning
360+x: A Panoptic Multi-modal Scene Understanding Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces 360+x, a multi-view, multi-modal scene understanding dataset (360° panoramic, third-person front, egocentric mono/binocular video plus audio, binaural delay, GPS and weather), and benchmarks five tasks with I3D, VGGish and ResNet-18 backbones.
Problem
Existing scene understanding datasets cover only one or a few viewpoints or modalities (e.g., egocentric or third-person only, visual-only), so they do not capture how humans combine views and modalities to perceive a scene.
Approach
A panoptic dataset that jointly captures a 360° panoramic camera (with motion-guided front-view crop), synchronized egocentric stereo/monocular videos, multi-channel audio with interaural-time-delay directional cues, location and weather metadata, and benchmarks multi-view and multi-modal fusion via hierarchical attention.
Key results
- Video classification: using all three views with V+A+D reaches 80.62% avg precision vs 51.95% for egocentric video only.
- Temporal action localisation with 360+x pre-trained extractors and V+A+D raises TriDet average mAP from 10.5 to 17.6 (baseline extractors with V only).
- Fine-tuning Kinetics400-pretrained I3D on 360+x improves THUMOS14 TAL average mAP by about 4.2 points over Kinetics400 alone (69.5 to 73.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 360° panoramic, third-person front, and egocentric views (ablation) | 360+x video scene classification · 360+x | Avg. Prec. (%): 63.73 -> 80.62 | Helped |
| add audio to video (ablation) | 360+x video scene classification with all three views · 360+x | Avg. Prec. (%): 63.73 -> 77.32 | Helped |
| add directional binaural delay to video and audio (ablation) | 360+x video scene classification with all three views · 360+x | Avg. Prec. (%): 77.32 -> 80.62 | Helped |
| add audio to video (ablation) | TriDet with baseline extractors · 360+x temporal action localisation | Avg. mAP: 10.5 -> 15.7 | Helped |
| use 360+x-pre-trained extractors instead of baseline extractors (ablation) | TriDet with video and audio · 360+x temporal action localisation | Avg. mAP: 15.7 -> 17.3 | Helped |
| co-train audio and directional binaural features (ablation) | audio-plus-directional query to video retrieval · 360+x cross-modality retrieval | R1 (%): 44.30 -> 55.88 | Helped |
| add video pace pre-training (ablation) | 360+x video scene classification with V + A + D · 360+x | Avg. Prec. (%): 80.62 -> 81.97 | Helped |
| add clip order pre-training (ablation) | 360+x video scene classification with V + A + D · 360+x | Avg. Prec. (%): 80.62 -> 82.18 | Helped |
| combine video pace and clip order pre-training (ablation) | 360+x video scene classification with V + A + D · 360+x | Avg. Prec. (%): 80.62 -> 83.32 | Helped |
| add video pace pre-training (ablation) | TriDet with V + A + D · 360+x temporal action localisation | Avg. mAP: 17.6 -> 18.8 | Helped |
| add clip order pre-training (ablation) | TriDet with V + A + D · 360+x temporal action localisation | Avg. mAP: 17.6 -> 18.5 | Helped |
| combine video pace and clip order pre-training (ablation) | TriDet with V + A + D · 360+x temporal action localisation | Avg. mAP: 17.6 -> 19.5 | Helped |
Surprising findings
- Self-supervised pretraining (video pace plus clip order) on 360+x outperforms supervised pretraining on the same data for temporal action localisation (average mAP 19.5 vs 17.6).
Limitations
- Dataset is relatively small (232 data examples, 2,152 videos) with only 28 scene categories.
- Directional audio and multi-view benefits are demonstrated only on a limited set of backbones (I3D, VGGish, ResNet-18).
Experimental setup
Datasets: 360+x Kinetics400 AudioSet THUMOS14
Benchmarks: 360+x THUMOS14
Metrics: average precision mAP@IoU recall@k
Baselines: ActionFormer TriDet TemporalMaxer Kinetics400 pre-trained I3D
Builds on: I3D VGGish ActionFormer TriDet TemporalMaxer
Closest papers by meaning
- Beyond Human Perception: Understanding Multi-Object World from Monocular View (CVPR 2025 · similarity 0.87)
- The 3D-PC: a benchmark for visual perspective taking in humans and machines (ICLR 2025 · similarity 0.86)
- PanoWorld: Towards Spatial Supersensing in 360◦ Panorama World (NeurIPS 2026 · similarity 0.86)
- ViewPoint: Panoramic Video Generation with Pretrained Diffusion Models (NeurIPS 2025 · similarity 0.86)
- VeXKD: The Versatile Integration of Cross-Modal Fusion and Knowledge Distillation for 3D Perception (NeurIPS 2024 · similarity 0.86)
- From an Image to a Scene: Learning to Imagine the World from a Million 360° Videos (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.