Research map / Video / Video-language understanding
H-MBA: Hierarchical MamBa Adaptation for Multi-Modal Video Understanding in Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes H-MBA, a plug-and-play hierarchical Mamba video adapter for Shikra-based MLLMs that models multi-granularity driving video context and fuses it via a query Mamba, achieving SOTA risk localization on DRAMA and strong captioning on BDD-X.
Problem
Multi-modal video understanding for autonomous driving with complex spatio-temporal motion, where existing MLLMs struggle with risk object localization and driving video captioning/justification.
Approach
Two-branch (low/high temporal resolution) hierarchy of Mamba blocks (T, DST, JST) produces multi-granularity video context, and a Q-Mamba adapter uses the current frame as a learnable query that cross-attends to these contexts and integrates them via latent Mamba, added residually to frozen CLIP features before the LLM.
Key results
- On DRAMA risk localization, H-MBA reaches 66.9% mIoU, a 5.5-point gain over the previous SOTA LCP (with optical flow) at 61.4%.
- On DRAMA captioning, H-MBA reports B4 0.536, CIDEr 3.227, the best among MLLM-based methods compared.
- On BDD-X, H-MBA gets description CIDEr 1.996 and justification CIDEr 0.949 among MLLM-based models; JST Mamba costs about 8.7 GFLOPs versus 116.9 for Timesformer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add H-MBA for video adaptation | Shikra · DRAMA risk localization | mIoU: 61.4 -> 66.9 | Helped |
| Add H-MBA for video understanding | Shikra · DRAMA caption | B4: 0.513 -> 0.536 | Helped |
| Add H-MBA for video understanding | Shikra · BDD-X full test set, Description | B4: 0.244 -> 0.301 | Helped |
| Add H-MBA for video understanding | Shikra · BDD-X full test set, Justification | B4: 0.072 -> 0.088 | Helped |
| Replace frame averaging with temporal sequence modeling (ablation) | Shikra on DRAMA · DRAMA detection | mIoU: 64.7 -> 66.9 | Helped |
| Use T-Mamba instead of a Transformer time-only block (ablation) | Shikra on DRAMA · DRAMA detection | mIoU: 65.6 -> 66.8 | Helped |
| Use DST Mamba instead of Timesformer (ablation) | Shikra on DRAMA · DRAMA detection | mIoU: 66.3 -> 66.6 | Helped |
| Use Q-Mamba instead of Direct Addition (ablation) | Shikra video adapter · DRAMA detection | mIoU: 66.6 -> 66.9 | Helped |
| Use Q-Mamba instead of Inception Concat (ablation) | Shikra video adapter · DRAMA detection | mIoU: 64.3 -> 66.9 | Helped |
| Use JST Mamba rather than the image-only baseline (ablation) | Shikra on DRAMA · DRAMA caption | B4: 0.513 -> 0.536 | Helped |
Surprising findings
- Direct temporal modeling with general video foundation models (e.g., Video-LLaMA, Video-Chat) underperforms the image-based Shikra baseline on DRAMA, which the authors attribute to domain bias.
- Inception-style concatenation adapter performs worse than the single-frame Shikra baseline on detection (mIoU 64.3 vs 64.0 only slightly higher, but lower caption scores than DA), suggesting difficulty converging with many concatenated features.
Limitations
- Evaluated only on DRAMA and BDD-X with visual input only; no comparison with stronger recent video MLLMs.
- Uses a fixed low-resolution input (224x224) with only 5 or 8 sampled frames.
- Some baseline numbers (e.g., HiLM-D caption results) are taken from different data splits, making comparisons partially inconsistent.
Experimental setup
Datasets: DRAMA BDD-X
Benchmarks: DRAMA BDD-X
Metrics: BLEU-4 METEOR ROUGE CIDEr SPICE mIoU Acc (IoU>0.5)
Baselines: Shikra BLIP-2 LLaVA Video-LLaMA Video-Chat HiLM-D LCP DriveGPT-4 ADAPT Timesformer
Builds on: Shikra Mamba Perceiver HiLM-D
Closest papers by meaning
- Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models (CVPR 2024 · similarity 0.87)
- M3Net: Multimodal Multi-task Learning for 3D Detection, Segmentation, and Occupancy Prediction in Autonomous Driving (AAAI 2025 · similarity 0.87)
- MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs (NeurIPS 2025 · similarity 0.86)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.86)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.86)
- MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.