Research map / Vision-language and multimodal / Multimodal LLMs
NAUTILUS: A Large Multimodal Model for Underwater Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds NautData (1.45M underwater instruction QA pairs over eight tasks) and Nautilus, LLaVA-1.5/Qwen2.5-VL extended with a physics-inspired vision feature enhancement module using depth and dark-pixel priors.
Problem
General LMMs underperform on underwater scene understanding due to domain shift from in-air data and image degradation from light scattering and absorption; there is no large multi-task underwater instruction-tuning dataset.
Approach
A plug-and-play vision feature enhancement module that restores degraded underwater visual features in feature space, guided by an underwater imaging model: it removes backscattering (via a dark-patch token and a learned global query) and compensates light absorption (via a depth-conditioned weight), with original and enhanced features fed to the LLM in parallel.
Key results
- Nautilus (Qwen2.5-VL) achieves best results on four of eight tasks on NautData, including fine-grained classification (93.8 acc), grounding (53.8 mIoU), detection (25.8 mAP) and image caption (0.223 METEOR).
- Nautilus (LLaVA-1.5) improves over LLaVA-1.5 on seven of eight tasks; on IOCfish5K counting, MAE drops from 59.2 to 51.2 for LLaVA-1.5 variant and from 31.6 to 30.9 for Qwen2.5-VL variant.
- Zero-shot grounding on MarineInst20M: Nautilus (Qwen2.5-VL) reaches 38.3 mIoU vs 38.0 for Qwen2.5-VL baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VFE to Qwen2.5-VL | Qwen2.5-VL · NautData test | fine-grained classification accuracy: 88.2 -> 93.8 (+5.6) | Helped |
| Add VFE to LLaVA-1.5 | LLaVA-1.5 · NautData test | grounding PR@0.5: 48.2 -> 52.2 (+4.0) | Helped |
| Add VFE to Qwen2.5-VL | Qwen2.5-VL · IOCfish5k test | accuracy: 70.8 -> 70.7(-0.1) | Hurt |
| Add VFE to Qwen2.5-VL | Qwen2.5-VL · MarineInst20M | grounding PR@0.5: 39.2 -> 39.6(+0.4) | Helped |
| Add depth encoder only (ablation) | Qwen2.5-VL baseline · NautData test | coarse classification accuracy: 87.9 -> 89.5 | Helped |
| Add depth-guided light-absorption restoration (ablation) | Qwen2.5-VL with depth encoder · NautData test | coarse classification accuracy: 89.5 -> 85.7 | Hurt |
| Add backscattering removal to VFE (ablation) | Qwen2.5-VL with light-absorption restoration · NautData test | coarse classification accuracy: 85.7 -> 90.0 | Helped |
| Pre-enhance images with Reti-Diff (ablation) | Qwen2.5-VL · NautData test | coarse classification accuracy: 87.9 -> 87.3 | Hurt |
| Pre-enhance images with SMDR-IS (ablation) | Qwen2.5-VL · NautData test | coarse classification accuracy: 87.9 -> 86.8 | Hurt |
| Pre-enhance images with CCL-Net (ablation) | Qwen2.5-VL · NautData test | coarse classification accuracy: 87.9 -> 82.5 | Hurt |
| Use image-feature-derived absorption weights instead of depth-feature-derived weights (ablation) | Qwen2.5-VL with VFE · NautData test | VQA METEOR: 0.367 -> 0.375 | Helped |
| Use depth-feature-derived absorption weights instead of image-feature-derived weights (ablation) | Qwen2.5-VL with VFE · NautData test | fine-grained classification accuracy: 90.2 -> 91.4 | Helped |
Surprising findings
- Simply adding a depth encoder degrades performance on four tasks, while feature-space enhancement with depth helps.
- Enhancing images with state-of-the-art underwater restoration methods before feeding them to the LMM consistently hurt performance on several tasks.
Limitations
- Does not address open-vocabulary or few-shot generalization to unseen species and categories.
- Multi-task optimization causes slight accuracy drop on counting single-choice questions.
Experimental setup
Datasets: NautData RUOD Deepfish Brackish FishNet IOCfish5K MarineInst20M
Benchmarks: NautData test set IOCfish5K test MarineInst20M
Metrics: accuracy METEOR mIoU PR@0.5 mAP mAP@0.5 MAE RMSE
Baselines: GPT-4o Qwen2.5-VL-72B Gemini 2.0 Flash MiniGPTv2 mPLUG-Owl3 InternVL-2.5 LLaVA-1.5 Reti-Diff SMDR-IS CCL-Net
Builds on: LLaVA-1.5 Qwen2.5-VL Depth Anything V2 dark channel prior
Closest papers by meaning
- Atlantis: Enabling Underwater Depth Estimation with Stable Diffusion (CVPR 2024 · similarity 0.87)
- Diving into Underwater: Segment Anything Model Guided Underwater Salient Instance Segmentation and A Large-scale Dataset (ICML 2024 · similarity 0.86)
- A Unified Image-Dense Annotation Generation Model for Underwater Scenes (CVPR 2025 · similarity 0.86)
- WaterDiffusion: Learning a Prior-involved Unrolling Diffusion for Joint Underwater Saliency Detection and Visual Restoration (AAAI 2025 · similarity 0.86)
- Synergistic Multiscale Detail Refinement via Intrinsic Supervision for Underwater Image Enhancement (AAAI 2024 · similarity 0.86)
- WebUOT-1M: Advancing Deep Underwater Object Tracking with A Million-Scale Benchmark (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.