Research map / 3D vision / Gaussian splatting
HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HUGS, a decomposed static/dynamic 3D Gaussian Splatting pipeline that jointly learns appearance, semantics, optical flow and unicycle-constrained object motion from posed RGB images and noisy pseudo-labels for holistic urban scene understanding.
Problem
Holistic understanding of dynamic urban scenes (novel view synthesis, 2D/3D semantics, object tracking) from RGB images alone, without LiDAR or ground-truth 3D bounding boxes, where noisy monocular detections make naive pose optimization unstable.
Approach
Decompose the scene into static Gaussians and per-vehicle dynamic Gaussians whose poses are parameterized by a unicycle model (learnable states and velocities with physical regularization), add semantic logits and flow rendering to 3D Gaussians, apply softmax in 3D before alpha blending to reduce floaters, and add a learned affine exposure correction.
Key results
- On dynamic scenes with noisy monocular boxes, HUGS reaches 28.20 PSNR / 0.919 SSIM on KITTI Scene06 versus 25.09 / 0.856 for MARS.
- On the KITTI-360 leaderboard, HUGS achieves 23.38 PSNR and 0.121 LPIPS for novel view synthesis, beating MARS (23.09, 0.174) and PNF (22.07 PSNR).
- Rendering runs at about 93 fps on a single RTX 4090 with roughly 30-minute training, while NSG and MARS run below 1 fps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HUGS instead of MARS | Dynamic scene novel view synthesis · KITTI Scene02 | PSNR: 23.30 -> 25.42 | Helped |
| Use HUGS instead of MARS | Dynamic scene novel view synthesis · KITTI Scene06 | PSNR: 25.09 -> 28.20 | Helped |
| Use HUGS instead of MARS | Dynamic scene novel view synthesis · vKITTI Scene02 | PSNR: 22.67 -> 26.21 | Helped |
| Use HUGS instead of MARS | Dynamic scene novel view synthesis · vKITTI Scene06 | PSNR: 21.67 -> 26.65 | Helped |
| Use HUGS instead of PNF | KITTI-360 novel-view semantic synthesis · KITTI-360 | mIoUcls: 73.06 -> 72.65 | Hurt |
| Use HUGS instead of Semantic Nerfacto | 3D semantic reconstruction · KITTI-360 | mIoUcls: 0.055 -> 0.505 | Helped |
| Add the unicycle model and optimize object poses rather than using noisy boxes directly (ablation) | HUGS dynamic scene reconstruction · KITTI 5% noise | PSNR: 23.83 -> 28.78 | Helped |
| Add the unicycle model to pose optimization (ablation) | HUGS dynamic scene reconstruction · KITTI 10% noise | PSNR: 22.75 -> 26.66 | Helped |
| Add the unicycle model to pose optimization (ablation) | HUGS dynamic scene reconstruction · KITTI 20% noise | PSNR: 20.56 -> 23.59 | Helped |
| Add the affine exposure transform (ablation) | HUGS appearance rendering · KITTI-360 static scenes | PSNR: 24.18 -> 24.52 | Helped |
| Add semantic loss (ablation) | HUGS appearance rendering · KITTI-360 static scenes | PSNR: 24.47 -> 24.52 | No clear effect |
| Add optical flow loss (ablation) | HUGS depth estimation · KITTI-360 static scenes | Depth: 1.031 -> 0.872 | Helped |
Surprising findings
- Applying softmax to 3D semantic logits before alpha blending, rather than to accumulated 2D logits, removes floaters and improves 3D semantic reconstruction.
- Semantic and flow losses contribute little to novel view synthesis quality, although flow supervision noticeably improves depth geometry.
Limitations
- Reconstructed dynamic objects can only rotate to a limited degree, so full object reconstruction needs category-level priors.
- The model lacks control over additional degrees of freedom such as lighting editing.
- Relies on noisy pseudo-labels from pre-trained monocular detectors, tracker, segmentation and flow models.
Experimental setup
Datasets: KITTI KITTI-360 Virtual KITTI 2
Benchmarks: KITTI Virtual KITTI 2 KITTI-360 leaderboard
Metrics: PSNR SSIM LPIPS mIoU chamfer distance rotation error translation error
Baselines: NSG MARS mip-NeRF PNF Semantic Nerfacto
Builds on: 3D Gaussian Splatting Panoptic Neural Fields Neural Scene Graphs Urban Radiance Field QD-3DT
Closest papers by meaning
- EVolSplat: Efficient Volume-based Gaussian Splatting for Urban View Synthesis (CVPR 2025 · similarity 0.89)
- SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splatting (NeurIPS 2025 · similarity 0.88)
- 3D StreetUnveiler with Semantic-aware 2DGS - a simple baseline (ICLR 2025 · similarity 0.88)
- CityGaussianV2: Efficient and Geometrically Accurate Reconstruction for Large-Scale Scenes (ICLR 2025 · similarity 0.88)
- Tackling View-Dependent Semantics in 3D Language Gaussian Splatting (ICML 2025 · similarity 0.88)
- FHGS: Feature-Homogenized Gaussian Splatting (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.