Research map / Detection and segmentation / 3D detection and driving
CodeMerge: Codebook-Guided Model Merging for Robust Test-Time Adaptation in Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CodeMerge, a codebook-guided model merging method for test-time adaptation of 3D perception in autonomous driving, using low-dimensional checkpoint fingerprints and ridge leverage scores to pick merge coefficients without repeated forward passes.
Problem
Test-time adaptation of 3D object detectors under distribution shifts (weather, sensor failures, cross-dataset) is unstable, and existing merging methods like MOS are computationally expensive because they load K checkpoints and run K forward passes.
Approach
Represent each fine-tuned checkpoint by a randomly projected fingerprint of the source model's penultimate features, store key-value pairs in a codebook, score checkpoints with ridge leverage scores (linked to inverse parameter-space curvature), and merge top-K with sign-consistent weighting.
Key results
- On nuScenes-C with SparseDrive, CodeMerge averages 0.2331 mAP vs 0.2028 for MOS and 0.1747 for no adaptation, with 0.4016 NDS.
- On Waymo→KITTI, CodeMerge reaches 84.62/66.31 AP_BEV/AP_3D (moderate), above MOS at 81.90/64.16.
- Compared with MOS on SparseDrive, CodeMerge reduces GPU memory by about 27% (39420 to 29868 MB) and runtime by 27.3% (37,619 s to 27,359 s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace no adaptation with CodeMerge | SparseDrive · nuScenes-C average | mAP: 0.1747 -> 0.2331 | Helped |
| replace MOS with CodeMerge | SparseDrive · nuScenes-C average | AMOTA: 0.1599 -> 0.1907 | Helped |
| apply CodeMerge instead of no adaptation | SparseDrive downstream modules · nuScenes-C average | mapping mAP: 0.2009 → 0.2859 | Helped |
| replace MOS with CodeMerge | SECOND · Waymo → KITTI | AP_3D: 64.16 -> 66.31 | Helped |
| replace no adaptation with CodeMerge | SECOND · KITTI-C mean | AP_3D: 54.74 -> 58.97 | Helped |
| use leverage-score checkpoint selection rather than random selection (ablation) | CodeMerge · nuScenes-C Motion | mAP: 0.2740 -> 0.2851 | Helped |
| use leverage-score checkpoint selection rather than selecting recent checkpoints (ablation) | CodeMerge · nuScenes-C Motion | mAP: 0.2480 -> 0.2851 | Helped |
| use leverage-score checkpoint selection rather than KMeans++ clustering (ablation) | CodeMerge · nuScenes-C Motion | mAP: 0.2746 -> 0.2851 | Helped |
| merge 5 checkpoints rather than 3 (ablation) | CodeMerge · nuScenes-C Motion | mAP: 0.2655 -> 0.2851 | Helped |
| merge 9 checkpoints rather than 5 (ablation) | CodeMerge · nuScenes-C Motion | mAP: 0.2851 vs. 0.2818 | No clear effect |
| reduce projection dimension from 1024 to 256 (ablation) | CodeMerge · nuScenes-C Motion | mAP: 0.2851 vs. 0.2749 | Hurt |
| increase projection dimension from 1024 to 2048 (ablation) | CodeMerge · nuScenes-C Motion | mAP: 0.2851 vs. 0.2799 | No clear effect |
Surprising findings
- Recent-checkpoint merging performs worst across tasks, indicating catastrophic forgetting when only the newest checkpoints are merged.
- Some TTA baselines (e.g., Tent) can degrade performance on certain corruptions relative to no adaptation.
Limitations
- Experiments are mainly on SparseDrive; UniAD and VAD reportedly degrade heavily on nuScenes-C, limiting adaptation.
- Only the 3D box regression head is updated for LiDAR detection; end-to-end setting freezes other components.
- Reported gains rely on a tuned set of hyperparameters (K=5, d'=1024) selected on the motion-blur ablation.
Experimental setup
Datasets: nuScenes nuScenes-C KITTI KITTI-C Waymo
Benchmarks: nuScenes-C KITTI-C KITTI Waymo nuScenes
Metrics: mAP NDS AMOTA AMOTP AP_BEV AP_3D mADE mFDE MR EPA
Baselines: No Adapt. SN ST3D Tent CoTTA SAR MemCLR Reg-TTA3D MOS DPO
Builds on: Model Synergy (MOS) Mean Teacher linear mode connectivity TIES-style sign-consistent merging
Closest papers by meaning
- DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation (CVPR 2025 · similarity 0.85)
- VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving (CVPR 2026 · similarity 0.85)
- MOS: Model Synergy for Test-Time Adaptation on LiDAR-Based 3D Object Detection (ICLR 2025 · similarity 0.85)
- Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving (NeurIPS 2024 · similarity 0.84)
- RLGF: Reinforcement Learning with Geometric Feedback for Autonomous Driving Video Generation (NeurIPS 2025 · similarity 0.84)
- DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.