Research map / Image generation / Editing and style transfer
Target Scanpath-Guided 360-Degree Image Enhancement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes target scanpath-guided 360° image enhancement, implemented as PSEM: clustering predicted scanpaths (TASSC), a DSDE-augmented SAM to select objects to remove, and a LoRA-fine-tuned Stable Diffusion to inpaint them.
Problem
Editing 360° images so that viewers' scanpaths (spatial and temporal gaze sequences) match a user-specified target scanpath, which existing saliency-guided 2D enhancement methods do not address.
Approach
Three-stage pipeline: a TASSC clustering algorithm (FastDTW + great-circle distance + complete linkage + FDBA barycenters) extracts a representative source scanpath; a Dual-Stream Scanpath Difference Encoder with relative temporal/spatial position encodings, plugged into SAM, selects two objects to edit; and an LoRA-fine-tuned Stable Diffusion v1-5 inpaints them.
Key results
- Enhanced images reduce LEV by 10.98% and DTW by 6.78% on average versus source images, and raise REC by 17.84% (42.19% with clustered representative scanpaths).
- TASSC clustering achieves a Silhouette coefficient of 0.47, Calinski-Harabasz index of 1345.92, and Davies-Bouldin index of 0.23.
- Using Inpainting Anything or LaMa instead of the fine-tuned SD gives similar scanpath similarity improvements (10.75% and 12.04%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| enhance the source images with PSEM | PSEM · 200 test images, average scanpaths | LEV: decreased by an average of 10.98% | Helped |
| enhance the source images with PSEM | PSEM · 200 test images, average scanpaths | DTW: decreased by 6.78% | Helped |
| enhance the source images with PSEM | PSEM · 200 test images, average scanpaths | REC: increased by 17.84% | Helped |
| enhance the source images with PSEM | PSEM · 200 test images, clustered representative scanpaths | LEV: 18.68% | Helped |
| enhance the source images with PSEM | PSEM · 200 test images, clustered representative scanpaths | DTW: 6.05% | Helped |
| enhance the source images with PSEM | PSEM · 200 test images, clustered representative scanpaths | REC: increased by 42.19% | Helped |
| fine-tune Stable Diffusion v1-5 with LoRA rather than use vanilla Stable Diffusion (ablation) | PSEM image generation · Image consistency and object removal | qualitative image quality | Helped |
| replace the fine-tuned Stable Diffusion inpainting model with Inpainting Anything (ablation) | PSEM · Scanpath modification | scanpath similarity improvement: 11.87% vs. 10.75% | Mixed |
| replace the fine-tuned Stable Diffusion inpainting model with LaMa (ablation) | PSEM · Scanpath modification | scanpath similarity improvement: 11.87% vs. 12.04% | Mixed |
Surprising findings
- Different inpainting backbones (IA, LaMa, fine-tuned SD) yield similar scanpath-guidance gains, suggesting inpainting quality matters little for attention guidance.
Limitations
- Training data relies on scanpaths predicted by a model rather than real human eye-tracking data.
- Editing is limited to removing up to two specific objects; pixel-level edits (contrast, color) are not supported.
- The editing stage does not account for the spherical properties of 360° images.
Experimental setup
Datasets: self-constructed 1000-pair 360-degree training set 200-image test set captured by the authors
Benchmarks: 200-image 360-degree test set
Metrics: Levenshtein distance dynamic time warping recurrence quantification analysis Silhouette coefficient Calinski-Harabasz index Davies-Bouldin index
Baselines: Inpainting Anything LaMa vanilla Stable Diffusion
Builds on: Segment Anything Stable Diffusion LoRA FastDTW ScanTD
Closest papers by meaning
- AuraFusion360: Augmented Unseen Region Alignment for Reference-based 360deg Unbounded Scene Inpainting (CVPR 2025 · similarity 0.82)
- Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation (NeurIPS 2024 · similarity 0.82)
- Three Cars Approaching within 100m! Enhancing Distant Geometry by Tri-Axis Voxel Scanning for Camera-based Semantic Scene Completion (CVPR 2025 · similarity 0.82)
- ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image (CVPR 2024 · similarity 0.81)
- Instruct 4D-to-4D: Editing 4D Scenes as Pseudo-3D Scenes Using 2D Diffusion (CVPR 2024 · similarity 0.81)
- Thinking in Boxes: 3D Editing in Real Images Made Easy (NeurIPS 2026 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.