Research map / 3D vision / 3D generation
OnlineSplatter: Pose-Free Online 3D Reconstruction for Free-Moving Objects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OnlineSplatter, a pose-free feed-forward model that builds object-centric 3D Gaussians online from monocular RGB frames using a dual-key (latent + direction) object memory with constant per-frame cost.
Problem
Reconstructing freely moving objects from monocular RGB video online, without camera poses, depth priors, or global bundle optimization, where static-scene pose-free methods fail.
Approach
A transformer that predicts pixel-aligned 3D Gaussians anchored to the first frame and reads from a bounded attention memory keyed by both learned latent features and an explicit orientation (direction) key, with usage/coverage-based sparsification.
Key results
- On GSO late-stage, OnlineSplatter reaches 31.737 PSNR / 0.969 SSIM / 0.075 LPIPS, versus best baseline NPS_dist3 at 24.141 PSNR.
- On HO3D late-stage, OnlineSplatter reaches 27.928 PSNR / 0.952 SSIM / 0.099 LPIPS, versus best baseline NPS_dist3 at 22.947 PSNR.
- Ablation on GSO shows removing the latent key drops late-stage M_avg from 0.810 to 0.596, and removing the direction key drops it to 0.723.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OnlineSplatter instead of the strongest listed baseline | OnlineSplatter vs. NoPoSplat and FreeSplatter · GSO Early-, Mid-, and Late-Stage | PSNR: 26.329 vs 23.331; 27.553 vs 23.757; 31.737 vs 24.141 | Helped |
| Use OnlineSplatter instead of the strongest listed baseline | OnlineSplatter vs. NoPoSplat · HO3D Early-, Mid-, and Late-Stage | PSNR: 23.627 vs 21.134; 25.803 vs 22.967; 27.928 vs 22.947 | Helped |
| add the latent key to memory (ablation) | OnlineSplatter · GSO Late-Stage | M_avg: 0.596 -> 0.810 | Helped |
| add the direction key to memory (ablation) | OnlineSplatter · GSO Late-Stage | M_avg: 0.723 -> 0.810 | Helped |
| encode memory from Transformer tokens rather than Gaussian parameters (ablation) | OnlineSplatter · GSO Late-Stage | M_avg: 0.611 -> 0.810 | Helped |
| use usage-and-coverage pruning instead of random pruning (ablation) | OnlineSplatter · GSO Late-Stage | M_avg: 0.764 -> 0.810 | Helped |
| use staged training rather than single-stage training (ablation) | OnlineSplatter · GSO Late-Stage | M_avg: 0.588 -> 0.810 | Helped |
| add ray alignment loss (ablation) | OnlineSplatter · GSO Late-Stage | M_avg: 0.682 -> 0.810 | Helped |
| add background penalty (ablation) | OnlineSplatter · GSO Late-Stage | M_avg: 0.795 -> 0.810 | Helped |
| add depth loss (ablation) | OnlineSplatter · GSO Late-Stage | M_avg: 0.805 -> 0.810 | Helped |
| use progressive frame sampling instead of sequential-only sampling (ablation) | OnlineSplatter · GSO Late-Stage | M_avg: 0.688 -> 0.810 | Helped |
| use progressive frame sampling instead of random-only sampling (ablation) | OnlineSplatter · GSO Late-Stage | M_avg: 0.764 -> 0.810 | Helped |
Surprising findings
- Removing the direction key leaves early and mid-stage metrics nearly unchanged (M_avg 0.699 and 0.701), with its effect appearing mainly in late stages.
Limitations
- Outputs 3D Gaussians, which are hard to convert to meshes for downstream use.
- Performance depends on quality of the initial reference view (occluded or blurry first frame hurts).
- Limited to rigid objects.
Experimental setup
Datasets: Objaverse GSO HO3D
Benchmarks: GSO HO3D
Metrics: PSNR SSIM LPIPS
Baselines: FreeSplatter-O (rand4) FreeSplatter-O (dist4) NoPoSplat (dist2) NoPoSplat (dist3) Fmov BundleSDF
Builds on: 3D Gaussian Splatting DINO XMem Orient Anything FreeSplatter NoPoSplat
Closest papers by meaning
- No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images (ICLR 2025 · similarity 0.89)
- SelfSplat: Pose-Free and 3D Prior-Free Generalizable 3D Gaussian Splatting (CVPR 2025 · similarity 0.89)
- ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos (CVPR 2025 · similarity 0.88)
- Free-Moving Object Reconstruction and Pose Estimation with Virtual Camera (AAAI 2025 · similarity 0.88)
- Motion Matters: Compact Gaussian Streaming for Free-Viewpoint Video Reconstruction (NeurIPS 2025 · similarity 0.88)
- FreeSplat: Generalizable 3D Gaussian Splatting Towards Free View Synthesis of Indoor Scenes (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.