Research map / Image generation / Editing and style transfer
Enhancing Visual Localization with Cross-Domain Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates cross-domain training images via a photometric-embedding 3DGS and a fine-tuned InstructPix2Pix editor to augment sparse nighttime data, then trains absolute pose regression (MS-T) with positional attention for cross-camera localization.
Problem
Absolute pose regression for visual localization degrades on cross-domain scenes, with multiple camera models (pinhole, fisheye, 360) and day/night appearance changes, where secondary domains are sparse (long-tail).
Approach
Combines a cross-domain 3D Gaussian Splatting with learnable photometric embeddings and dynamic-object suppression, a text-guided image editor fine-tuned on temporal transformation data to augment sparse secondary domains, anchor-based pose sampling for generation, and positional attention to align cross-camera image features.
Key results
- Outperforms the previous best MS-T-VC2 by an average of 51% during the day and 67% at night on 360Loc, and the paper reports an overall average improvement of 59% across domains.
- Cross-domain 3DGS raises the day PSNR from 22.38 (Scaffold-GS) to 24.64 and the night PSNR to 26.07 on the Atrium scene.
- Fine-tuning the editor raises SSIM from 0.338 to 0.709 and lowers LPIPS from 0.428 to 0.337 relative to the pretrained model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use cross-domain 3DGS instead of Scaffold-GS (ablation) | 3D scene reconstruction on Atrium datasets · Atrium Day | PSNR: 22.38 -> 24.64 | Helped |
| Use cross-domain 3DGS instead of Scaffold-GS (ablation) | 3D scene reconstruction on Atrium datasets · Atrium Day | LPIPS: 0.38 -> 0.23 | Helped |
| Use cross-domain 3DGS rather than Scaffold-GS (ablation) | 3D scene reconstruction · Atrium Night | PSNR | Helped |
| Use cross-domain 3DGS rather than Scaffold-GS (ablation) | 3D scene reconstruction · Atrium Night | LPIPS | Helped |
| Fine-tune the image editing model for the task (ablation) | image editing model on Atrium · Atrium | SSIM: 0.338 -> 0.709 | Helped |
| Fine-tune the image editing model for the task (ablation) | image editing model on Atrium · Atrium | LPIPS: 0.428 -> 0.337 | Helped |
| Add positional attention (ablation) | visual localization model · Primary datasets (Average) | median translation/orientation error (m, °): 2.8/14.6 -> 2.0/10.8 | Helped |
| Generate training data with the proposed method rather than MS-T-VC2 | cross-domain visual localization · Atrium Day (Average) | median translation/orientation error (m, °): 6.5/39.0 -> 2.6/14.5 | Helped |
| Generate training data with the proposed method rather than MS-T-VC2 | cross-domain visual localization · Concourse Day (Average) | median translation/orientation error (m, °): 2.1/14.3 -> 1.5/6.4 | Helped |
| Generate training data with the proposed method rather than MS-T-VC2 | cross-domain visual localization · Hall Day (Average) | median translation/orientation error (m, °): 2.4/28.2 -> 1.2/10.2 | Helped |
| Generate training data with the proposed method rather than MS-T-VC2 | cross-domain visual localization · Piatrium Day (Average) | median translation/orientation error (m, °): 4.6/25.4 -> 2.8/12.0 | Helped |
| Generate training data with the proposed method rather than MS-T-VC2 | cross-domain visual localization · Atrium Night (Average) | median translation/orientation error (m, °): 9.5/64.9 -> 2.2/12.8 | Helped |
Surprising findings
- Pinhole queries (narrow FOV) localize markedly worse than the wide-FOV cameras even with augmentation, which the authors attribute to the inherent FOV limitation.
Limitations
- Significant geometric changes in the scene over time can impair localization performance.
- Narrow-FOV pinhole queries remain a performance bottleneck.
- Hallucination noise from the editing model requires a two-stage training strategy to mitigate.
Experimental setup
Datasets: 360Loc
Benchmarks: 360Loc
Metrics: median translation error (m) median orientation error (deg) PSNR LPIPS SSIM
Baselines: PoseNet (PN) MS-T PN-VC2 MS-T-VC2 Scaffold-GS
Builds on: 3D Gaussian Splatting Scaffold-GS InstructPix2Pix NeRF-W MS-T
Closest papers by meaning
- RELOCATE: A Simple Training-Free Baseline for Visual Query Localization Using Region-Based Representations (CVPR 2025 · similarity 0.88)
- Aligning Geometric Spatial Layout in Cross-View Geo-Localization via Feature Recombination (AAAI 2024 · similarity 0.87)
- ConDo: Continual Domain Expansion for Absolute Pose Regression (AAAI 2025 · similarity 0.87)
- From Coarse to Fine: A Matching and Alignment Framework for Unsupervised Cross-View Geo-Localization (AAAI 2025 · similarity 0.87)
- R-SCoRe: Revisiting Scene Coordinate Regression for Robust Large-Scale Visual Localization (CVPR 2025 · similarity 0.86)
- 360Loc: A Dataset and Benchmark for Omnidirectional Visual Localization with Cross-device Queries (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.