Research map / Video / Video generation
TC-Light: Temporally Coherent Generative Rendering for Realistic World Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free long-video relighting that inflates IC-Light with VidToMe token merging and multi-axis (xy/yt) decayed denoising, then refines the result with a two-stage post-optimization (per-frame exposure embedding, then a Unique Video Tensor canonical representation).
Problem
Video relighting/texture re-rendering of long, highly dynamic videos suffers from temporal flicker and high computational cost, while existing methods are limited to portrait domains or run out of memory on long clips.
Approach
Zero-shot inflation of an image relighting diffusion model (IC-Light) using token merging and decayed multi-axis denoising, followed by an optimization-based two-stage refinement over a compact canonical tensor (UVT) indexed by optical flow ID and quantized color (plus optional depth voxels) to enforce temporal consistency efficiently.
Key results
- On the 58-video benchmark, Ours-full reaches 97.80% Motion-S and 91.75 WarpSSIM versus 94.52% and 71.22 for per-frame IC-Light, with 23.96% user preference rate.
- On Virtual KITTI 2, Ours-full achieves SSIM 0.5910 and LPIPS 0.3971 versus 0.4833/0.4841 for Cosmos-Transfer1, at about 1002 s per clip.
- Ours-light adds about 2.4 minutes and 2.7 GB VRAM over the VidToMe baseline, while Light-A-Video, RelightVid and VideoDirector run out of memory on long videos.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add first-stage exposure alignment (ablation) | TC-Light on VidToMe · CARLA and InteriorNet subset | WarpSSIM: 77.60 -> 81.29 | Helped |
| Add second-stage video-frame optimization (ablation) | TC-Light on VidToMe · CARLA and InteriorNet subset | WarpSSIM: 81.29 -> 90.58 | Helped |
| Use UVT instead of optimizing full video frames in stage two (ablation) | TC-Light · CARLA and InteriorNet subset | WarpSSIM: 90.58 -> 91.04 | Helped |
| Replace hard mask with soft mask (ablation) | TC-Light with UVT · CARLA and InteriorNet subset | WarpSSIM: 91.04 -> 91.05 | Helped |
| Initialize UVT optimization from first-stage results rather than from scratch (ablation) | TC-Light · CARLA and InteriorNet subset | WarpSSIM: 90.65 -> 91.04 | Helped |
| Add depth cues to the UVT representation (ablation) | TC-Light with UVT · CARLA and InteriorNet subset | WarpSSIM: 91.05 -> 91.12 | Helped |
| Add instance-segmentation cues to the UVT representation (ablation) | TC-Light with UVT · CARLA and InteriorNet subset | CLIP-T: 0.2868 -> 0.2851 | Hurt |
| Add multi-axis denoising (ablation) | TC-Light diffusion module · CARLA and InteriorNet subset | WarpSSIM: 91.12 -> 95.52 | Helped |
| Align noise statistics with AIN (ablation) | TC-Light diffusion module · CARLA and InteriorNet subset | WarpSSIM: 93.74 -> 95.44 | Helped |
| Apply decayed noise weighting (ablation) | TC-Light diffusion module · CARLA and InteriorNet subset | WarpSSIM: 91.12 -> 93.74 | Helped |
| Downscale NavSim videos from 960×536 to 480×264 | TC-Light · NavSim subset | Warp-SSIM: 90.46 -> 88.36 | Hurt |
Surprising findings
- Light-A-Video, RelightVid and VideoDirector ran out of memory on all long benchmark videos, so they could not be compared directly.
- Downsampling NavSim input by 4x to simulate fast motion reduced Warp-SSIM by about 5% while CLIP-T and Motion-S changed less than 2%.
Limitations
- Inherits IC-Light limitations: hard shadows and large edits on low-light inputs are hard, and performance drops below 512 resolution.
- Optical-flow errors cause artifacts in textureless regions and under very fast motion.
- The temporal consistency loss can smooth texture in flickering areas, sacrificing some detail.
Experimental setup
Datasets: SceneFlow CARLA Waymo NavSim AgiBot Digital World DROID InteriorNet SCAND DRONE (self-collected) Virtual KITTI 2
Benchmarks: long highly dynamic video relighting benchmark (58 videos) Virtual KITTI 2
Metrics: Motion-S WarpSSIM CLIP-T User-PF (Bradley-Terry) SSIM LPIPS FPS Time VRAM
Baselines: IC-Light (per-frame) VidToMe Slicedit Cosmos-Transfer1 Light-A-Video RelightVid VideoDirector
Builds on: IC-Light VidToMe Slicedit MemFlow
Closest papers by meaning
- UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting (NeurIPS 2025 · similarity 0.88)
- WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perception (NeurIPS 2025 · similarity 0.87)
- Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset (CVPR 2025 · similarity 0.87)
- IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation (NeurIPS 2025 · similarity 0.86)
- Coarse-to-Real: Generative Rendering for Populated Dynamic Scenes (NeurIPS 2026 · similarity 0.86)
- LightIt: Illumination Modeling and Control for Diffusion Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.