Research map / Detection and segmentation / Open-set and few-shot detection
Leader360V: A Large-scale, Real-world 360 Video Dataset for Multi-task Learning in Diverse Environment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Leader360V, a 10K+ real-world 360 video dataset with segmentation and tracking annotations, built with A^3360V, an automatic pipeline combining 2D segmentors, SAM2 and LLM checkers with human revision.
Problem
Large-scale labelled real-world 360 video datasets for instance segmentation and tracking are lacking, and annotating distorted equirectangular video is costly and complex.
Approach
A three-phase annotation pipeline (Initial Annotation with SDR module, Auto-Refine with MCR module, Manual Revise) that coordinates pre-trained 2D segmentors (CropFormer, OneFormer), SAM2 mask prompting with shift-voting for distortion, and GPT-4o as semantic/blank-area/annotation checkers to automate 360 video labelling.
Key results
- Fine-tuning SwinB-AOT-L on the Leader360V training split raises J&F from 38.8 (untrained) to 58.3, and PSCFormer rises from 24.3 to 60.6 J&F.
- Phase II MCR refinement raises J&F of updated frames from 35.8 (SAM2) to 75.9 on 100 sampled frames.
- Annotating one video at 1 fps takes over 20 hours manually versus about 2 hours with the pipeline, per the paper's estimate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train SwinB-AOT-L on the Leader360V train subset | SwinB-AOT-L · Leader360V-S Test | J & F: 38.8 -> 58.3 (+19.5) | Helped |
| Train PSCFormer on the Leader360V train subset | PSCFormer · Leader360V-S Test | J & F: 24.3 -> 60.6 (+36.3) | Helped |
| Train SimTrack on the Leader360V train subset | SimTrack · Leader360V-S Test | S_dual: 0.291 -> 0.417 (+12.6) | Helped |
| Add SDR after the 2D segmentor (ablation) | A³360V Phase I · Updated Frame | J & F: 25.1 -> 67.3 (+42.2) | Helped |
| Add MCR to the SAM2 output (ablation) | A³360V Phase II · Updated Frame | J & F: 35.8 -> 75.9 (+40.1) | Helped |
| Add Leader360V self-collected training data | AOT · YouTubeVOS Test | J & F: 72.8 -> 61.4 (↓11.4) | Hurt |
| Add Leader360V self-collected training data | AOT · Leader360V Test | J & F: 42.9 -> 51.6 (↑8.7) | Helped |
| Fine-tune on Leader360V after training on YouTubeVOS | XMem · Leader360V Test | J & F: 42.5 -> 55.1 (↑12.6) | Helped |
| Fine-tune on Leader360V after training on TrackingNet | SimTrack · Leader360V Test | S_dual: 29.1 -> 41.5 (↑12.4) | Helped |
| Fine-tune on Leader360V after training on TrackingNet | SimTrack · TrackingNet Test | S_dual: 83.1 -> 63.3 (↓19.8) | Hurt |
Surprising findings
- Off-the-shelf SAM-based and 360-specific models (PerSAM, SAM-PT, GoodSAM) perform poorly on 360 video, and the 360-specific tracker SiamX-360 does not meet expectations.
- Models trained on YouTubeVOS lose 10-18 J&F points when applied directly to Leader360V, and VOT models degrade even more.
Limitations
- Dataset does not cover all object classes common in daily life.
- Does not include annotations for motion states of moving objects.
- Pipeline fails near the poles/zenith and nadir, for thin objects like railings, and on objects near the equator.
Experimental setup
Datasets: 360VOTS PanoVOS WEB360 360+x YouTube360 YouTubeVOS TrackingNet COCO ADE20K Cityscapes
Benchmarks: Leader360V-S Test Leader360V-T Test YouTubeVOS Test TrackingNet Test
Metrics: J&F J F S_dual P_dual user study scores (M, W, D)
Baselines: PerSAM SAM-PT GoodSAM XMem AOT PSCFormer SimTrack SiamX AiATrack ProContEXT MixFormer STCN
Builds on: SAM2 CropFormer OneFormer GPT-4o
Closest papers by meaning
- From an Image to a Scene: Learning to Imagine the World from a Million 360° Videos (NeurIPS 2024 · similarity 0.88)
- TAPVid-360: Tracking Any Point in 360 from Narrow Field of View Video (NeurIPS 2025 · similarity 0.87)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.86)
- MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views (NeurIPS 2024 · similarity 0.86)
- GMOS: Grounding Moving Object Segmentation in 3D Space and Time (NeurIPS 2026 · similarity 0.85)
- Unleashing Multispectral Video's Potential in Semantic Segmentation: A Semi-supervised Viewpoint and New UAV-View Benchmark (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.