Research map / Vision-language and multimodal / Human and embodied modeling
Structural Information Guided Multimodal Pre-training for Vehicle-Centric Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VehicleMAE, a masked autoencoder pre-training framework for vehicle images guided by BDCN sketch structure priors and CLIP semantic priors, pre-trained on a new 1M-image Autobot1M dataset.
Problem
Vehicle-centric perception models are typically pre-trained on generic classification data, ignoring vehicle-specific structure (outline lines/curves) and available text descriptions, which can yield sub-optimal downstream performance.
Approach
Extends MAE with two priors: a structural prior that distills features from BDCN-extracted vehicle contour maps into the masked-token reconstruction, and a semantic prior that uses frozen CLIP image/text encoders to enforce feature alignment and similarity-distribution consistency with vehicle text descriptions.
Key results
- On VAR (VeRi-776), VehicleMAE reaches 92.21% mA vs 90.19% for MAE pre-trained on Autobot1M and 89.69% for MAE on ImageNet-1K.
- On V-ReID, VehicleMAE achieves 85.6% mAP and 97.9% R1, versus 75.5% mAP and 95.4% R1 for MAE on Autobot1M.
- With 10% of downstream training data on part segmentation, VehicleMAE reaches 65.09% mIoU vs 62.35% for MAE on Autobot1M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add structural patch reconstruction loss L_mim (ablation) | MAE pre-trained on Autobot1M · VeRi vehicle attribute recognition | mA: 90.19 -> 91.27 | Helped |
| Add structural CLS loss L_cls alongside L_mim (ablation) | MAE pre-trained on Autobot1M · VeRi vehicle attribute recognition | mA: 91.27 -> 91.71 | Helped |
| Add semantic consistency loss L_cs (ablation) | MAE pre-trained on Autobot1M · VeRi vehicle re-identification | mAP: 75.5 -> 84.1 | Helped |
| Add semantic feature loss L_cf alongside L_cs (ablation) | MAE pre-trained on Autobot1M · VeRi vehicle re-identification | mAP: 84.1 -> 85.2 | Helped |
| Use the full structural and semantic losses (ablation) | MAE pre-trained on Autobot1M · VeRi vehicle re-identification | mAP: 75.5 -> 85.6 | Helped |
| Set masked-token ratio to 0.75 rather than 0.25, 0.50, or 0.85 (ablation) | VehicleMAE · VeRi vehicle attribute recognition | mA: 90.48; 91.88; 92.21; 90.73 | Mixed |
| Pre-train MAE on Autobot1M instead of ImageNet-1K | MAE · VeRi vehicle attribute recognition | mA: 89.69 -> 90.19 | Helped |
| Use VehicleMAE instead of the strongest reported baseline | ViT-base · VeRi vehicle attribute recognition | mA: 90.38 -> 92.21 | Helped |
| Use VehicleMAE instead of MAE pre-trained on ImageNet-1K | ViT-base · VeRi vehicle re-identification | mAP: 76.7 -> 85.6 | Helped |
| Use VehicleMAE instead of MoCov3 | ViT-base · Stanford Cars | Accuracy: 91.3 -> 94.5 | Helped |
| Use VehicleMAE instead of MoCov3 | ViT-base · PartImageNet vehicle part segmentation | mIoU: 73.17 -> 73.29 | Helped |
Surprising findings
- Authors state VehicleMAE gives no significant improvement for vehicle detection when many background objects are present in large scenes.
Limitations
- Does not improve vehicle detection in large scenes with many background objects, per the authors.
- Reported gains on some tasks are small (e.g., MAE on Autobot1M is roughly on par with ImageNet MAE on several metrics).
Experimental setup
Datasets: Autobot1M CompCars VERI-Wild VeRi-776 Stanford Cars PartImageNet
Benchmarks: VeRi-776 Stanford Cars PartImageNet
Metrics: mA accuracy precision recall F1 mAP R1 mIoU mAcc
Baselines: Scratch MoCov3 DINO iBOT MAE (ImageNet-1K) MAE (Autobot1M) VTB TransReID TransFG SETR
Builds on: MAE CLIP BDCN TransReID SETR
Closest papers by meaning
- RoadSceneBench: A Lightweight Benchmark for Mid-Level Road Scene Understanding (CVPR 2026 · similarity 0.86)
- Infer the Whole from a Glimpse of a Part: Keypoint-Based Knowledge Graph for Vehicle Re-Identification (AAAI 2025 · similarity 0.85)
- CLIP-driven View-aware Prompt Learning for Unsupervised Vehicle Re-identification (AAAI 2025 · similarity 0.85)
- MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding (CVPR 2024 · similarity 0.85)
- DriveGPT4-V2: Harnessing Large Language Model Capabilities for Enhanced Closed-Loop Autonomous Driving (CVPR 2025 · similarity 0.85)
- Weakly Supervised Semantic Segmentation for Driving Scenes (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.