Research map / Vision-language and multimodal / Human and embodied modeling
MMTL-UniAD: A Unified Framework for Multimodal and Multi-Task Learning in Assistive Driving Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MMTL-UniAD, a multimodal multi-task network (multi-axis region attention encoder plus dual-branch shared/specific embedding) that jointly recognizes driver emotion, driver behavior, traffic context and vehicle behavior on the AIDE dataset.
Problem
Assistive driving perception needs joint recognition of driver states and traffic context, but existing multi-task models suffer negative transfer between weakly related tasks and rely on single-modal inputs.
Approach
Combines horizontal-vertical attention with region attention (MARNet) to select task-relevant multi-view features, and a dual-branch embedding that adaptively mixes task-shared and task-specific features via learnable weights to reduce negative transfer.
Key results
- MMTL-UniAD reaches 82.30 mAcc on AIDE, versus 78.20 for the best listed baseline (ShuffleNet-V1), per Table 1.
- Driver behavior accuracy improves by 4.64% and vehicle behavior by 3.62% over baselines.
- Removing MARNet or the dual-branch embedding lowers mAcc by 5.34%-12.05%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MMTL-UniAD instead of the strongest baseline | AIDE · AIDE test, DER | accuracy: 74.87 -> 76.67 | Helped |
| use MMTL-UniAD instead of the strongest baseline | AIDE · AIDE test, DBR | accuracy: 68.97 -> 73.61 | Helped |
| use MMTL-UniAD instead of the strongest baseline | AIDE · AIDE test, TCR | accuracy: 93.75 -> 93.91 | Helped |
| use MMTL-UniAD instead of the strongest baseline | AIDE · AIDE test, VBR | accuracy: 81.21 -> 85.00 | Helped |
| use MMTL-UniAD instead of the strongest baseline | AIDE · AIDE test | mAcc: 78.20 -> 82.30 | Helped |
| add MARNet and dual-branch multimodal embedding instead of VGG and concatenation (ablation) | MMTL-UniAD · AIDE test | mAcc: 70.25 -> 82.30 | Helped |
| add MARNet instead of replacing it with VGG (ablation) | MMTL-UniAD · AIDE test | mAcc: 76.79 -> 82.30 | Helped |
| add dual-branch multimodal embedding instead of concatenation (ablation) | MMTL-UniAD · AIDE test | mAcc: 76.96 -> 82.30 | Helped |
| combine driver face/body, joint, and scene modalities instead of using face/body only (ablation) | MMTL-UniAD · AIDE test | mAcc: 76.91 -> 82.30 | Helped |
| combine driver face/body, joint, and scene modalities instead of using gesture/posture only (ablation) | MMTL-UniAD · AIDE test | mAcc: 55.69 -> 82.30 | Helped |
| combine driver face/body, joint, and scene modalities instead of using scene images only (ablation) | MMTL-UniAD · AIDE test | mAcc: 78.13 -> 82.30 | Helped |
| jointly train driver-state and traffic-context tasks instead of driver-state tasks only (ablation) | MMTL-UniAD · AIDE test, DER | accuracy: 72.22 -> 76.67 | Helped |
Surprising findings
- Single-modality input (e.g., only multi-view scene images) drops mAcc sharply to 55.69, and driver-state-only joint training reduces DER and DBR accuracy.
Limitations
- Evaluated on a single small dataset (AIDE, 2,898 samples)
- Relies on multi-view and joint pose/gesture inputs that may be costly to collect in practice
Experimental setup
Datasets: AIDE
Benchmarks: AIDE
Metrics: accuracy mAcc
Baselines: ResNet-18 ResNet-34 VGG16 CMT GroupMixFormer AbSViT GLMDriveNet TimeSFormer Video Swin Transformer ShuffleNet-V1 I3D SlowFast
Builds on: Video Swin Transformer ResNet soft parameter sharing
Closest papers by meaning
- M3Net: Multimodal Multi-task Learning for 3D Detection, Segmentation, and Occupancy Prediction in Autonomous Driving (AAAI 2025 · similarity 0.87)
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.84)
- SalM²: An Extremely Lightweight Saliency Mamba Model for Real-Time Cognitive Awareness of Driver Attention (AAAI 2025 · similarity 0.84)
- Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models (CVPR 2024 · similarity 0.84)
- SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models (ICML 2025 · similarity 0.84)
- H-MBA: Hierarchical MamBa Adaptation for Multi-Modal Video Understanding in Autonomous Driving (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.