Research map / Point clouds / Segmentation and detection
TREND: Unsupervised 3D Representation Learning via Temporal Forecasting for LiDAR Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TREND, an unsupervised LiDAR 3D representation learning method that pretrains a sparse 3D encoder by forecasting future LiDAR observations via ego-motion-conditioned recurrent embedding and a temporal LiDAR neural field rendered with SDF.
Problem
Unsupervised 3D representation learning for LiDAR perception relies on masked autoencoding or contrastive learning on single frames, ignoring temporal LiDAR sequences that encode object motion and semantics.
Approach
Pretrain the 3D encoder by temporal forecasting: a recurrent embedding conditioned on sinusoidally encoded ego-motion predicts future scene features, which are decoded via a temporal LiDAR neural field (SDF-based differentiable ray rendering with timestamp and intensity) to reconstruct future LiDAR returns.
Key results
- TREND reaches 33.17 mAP and 46.21 NDS on NuScenes fine-tuning versus 31.06 mAP and 44.75 NDS from scratch, outperforming Copilot4D (31.08 mAP) and ViDAR* (30.76 mAP).
- On Once at 100% downstream data, TREND improves over from-scratch by 1.77% absolute, and the authors report about 4x the gain of previous SOTA pretraining methods (+1.06% vs +0.22%).
- Compared with 2D-prior methods, TREND (33.17 mAP) slightly exceeds Sonata (32.32) and LiMoE first stage (32.21) on NuScenes while using only LiDAR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TREND instead of Copilot4D pretraining | NuScenes LiDAR detection · NuScenes | mAP: 31.08 -> 33.17 | Helped |
| Use TREND instead of Copilot4D pretraining | NuScenes LiDAR detection · NuScenes | NDS: 45.05 -> 46.21 | Helped |
| Use TREND instead of Sonata pretraining | NuScenes LiDAR detection · NuScenes | mAP: 32.32 -> 33.17 | Helped |
| Use TREND instead of Sonata pretraining | NuScenes LiDAR detection · NuScenes | NDS: 46.07 -> 46.21 | Helped |
| Add height filtering before uniform ray sampling (ablation) | TREND · NuScenes | mAP: 31.68 -> 33.17 | Helped |
| Add height filtering before uniform ray sampling (ablation) | TREND · NuScenes | NDS: 45.77 -> 46.21 | Helped |
| Use uniform rather than farthest point sampling with height filtering (ablation) | TREND · NuScenes | mAP: 32.52 -> 33.17 | Helped |
| Use uniform rather than farthest point sampling with height filtering (ablation) | TREND · NuScenes | NDS: 46.48 -> 46.21 | Hurt |
| Use height-filtered FPS instead of FPS without height filtering (ablation) | TREND ray sampling · NuScenes | mAP: 31.65 -> 32.52 | Helped |
| Use height-filtered FPS instead of FPS without height filtering (ablation) | TREND ray sampling · NuScenes | NDS: 45.42 -> 46.48 | Helped |
Surprising findings
- Naively applying occupancy-based forecasting (ViDAR, Copilot4D) yields minimal or negative gains over from-scratch initialization.
- Ground-point filtering matters more than the choice of ray sampling strategy (FPS vs uniform).
Limitations
- Pretraining is about 8% slower per epoch than prior methods (65 vs 60 min on 8 A100) and adds a neural field rendering pipeline.
- Limited pedestrian performance in the 5% low-data regime, attributed to cylinder-like geometry.
- Smaller improvement when transferring from Once to Waymo than in-domain.
Experimental setup
Datasets: Once NuScenes SemanticKITTI Waymo
Benchmarks: Once NuScenes SemanticKITTI Waymo
Metrics: mAP NDS
Baselines: From-scratch Copilot4D ViDAR LiMoE Sonata SLiDR
Builds on: masked autoencoding contrastive learning Copilot4D ViDAR
Closest papers by meaning
- CO3: Cooperative Unsupervised 3D Representation Learning for Autonomous Driving (ICLR 2023 · similarity 0.90)
- Temporal Consistent 3D LiDAR Representation Learning for Semantic Perception in Autonomous Driving (CVPR 2023 · similarity 0.90)
- LSK3DNet: Towards Effective and Efficient 3D Perception with Large Sparse Kernels (CVPR 2024 · similarity 0.88)
- HiLoTs: High-Low Temporal Sensitive Representation Learning for Semi-Supervised LiDAR Segmentation in Autonomous Driving (CVPR 2025 · similarity 0.88)
- Fine-grained Image-to-LiDAR Contrastive Distillation with Visual Foundation Models (NeurIPS 2024 · similarity 0.88)
- 3D LiDAR Mapping in Dynamic Environments using a 4D Implicit Neural Representation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.