Research map / Video / Video-language understanding
Fire360: A Benchmark for Robust Perception and Episodic Memory in Degraded 360° Firefighting Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Fire360, a benchmark of 228 real 360° firefighter training videos with annotations for five tasks (VQA, temporal action captioning, object localization, safety reasoning, and transformed object retrieval), evaluated zero-shot on several VLMs.
Problem
Vision-language models lack robust perception and episodic memory under real-world degradation (smoke, low light, thermal distortion, structural deformation) in safety-critical firefighting scenes, which existing clean or synthetic benchmarks do not capture.
Approach
A real-world 360° firefighting video benchmark with degradation metadata and a new Transformed Object Retrieval task that matches pristine exemplars to fire-damaged objects in unpaired scenes.
Key results
- GPT-4o reaches 39.8% top-1 accuracy on Transformed Object Retrieval versus 83.5% for human experts.
- GPT-4o scores 53.8% on 360° VQA versus 91.4% for humans; GLaMM-7B achieves 0.341 BLEU-4 on temporal action captioning versus 0.85 human agreement.
- Grounding DINO localization mean IoU is 38.4% overall, dropping to 22.9% in low-visibility scenes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use rectilinear rather than equirectangular input (ablation) | GPT-4o · Fire360 VQA | Top-1 Accuracy: 53.8% -> 62.4% | Helped |
| Use rectilinear rather than equirectangular input (ablation) | Qwen-VL · Fire360 VQA | Top-1 Accuracy: 47.2% -> 55.6% | Helped |
| Use rectilinear rather than equirectangular input (ablation) | LLaVA-v1.5-13B · Fire360 VQA | Top-1 Accuracy: 50.3% -> 58.9% | Helped |
| Use rectilinear rather than equirectangular input (ablation) | BLIP-2 (OPT-6.7B) · Fire360 VQA | Top-1 Accuracy: 42.7% -> 48.2% | Helped |
| Use rectilinear projection rather than equirectangular input (ablation) | Grounding DINO · Fire360 Object Localization | Mean IoU: 38.4% -> 47.1% | Helped |
| Evaluate in low-visibility rather than clean conditions | Grounding DINO · Fire360 Object Localization | Mean IoU: 68.2% -> 22.9% | Hurt |
| Evaluate under high degradation | GPT-4o · Fire360 VQA | Top-1 Accuracy: 53.8% -> 9.8% | Hurt |
| Use Qwen-VL instead of GPT-4o (Prompted) | Fire360 Safety-Critical Reasoning · Fire360 Safety-Critical Reasoning | Checklist Accuracy: 28.9% -> 32.5% | Helped |
Surprising findings
- Rectilinear projection improves model accuracy across VQA and localization compared with equirectangular inputs.
- Qwen-VL slightly outperforms GPT-4o on safety-critical reasoning (32.5% vs 28.9% checklist accuracy).
Limitations
- Single firefighter training institute recording limits generalization; only 43.9% indoor scenes.
- Human agreement ceilings may underestimate performance limits due to projection distortion in ambiguous cases.
- TOR currently lacks multi-frame temporal modeling.
Experimental setup
Datasets: Fire360
Benchmarks: Fire360
Metrics: top-1 accuracy exact match accuracy BLEU-4 mean IoU checklist accuracy
Baselines: GPT-4o Qwen-VL LLaVA-v1.5-13B BLIP-2 CLIP GLaMM-7B Grounding DINO
Builds on: Grounding DINO CLIP BLIP-2
Closest papers by meaning
- DetectiumFire: A Comprehensive Multi-modal Dataset Bridging Vision and Language for Fire Understanding (NeurIPS 2025 · similarity 0.88)
- Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events (CVPR 2025 · similarity 0.86)
- VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.84)
- MVVBench: Benchmarking 4D Reasoning in Vision-Language Models (NeurIPS 2026 · similarity 0.84)
- SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models (ICML 2026 · similarity 0.84)
- Act2See: Emergent Active Visual Perception for Video Reasoning (CVPR 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.