Research map / Image generation / Editing and style transfer

StyLitGAN: Image-Based Relighting via Latent Control

CVPR 2024 · Image generation › Editing and style transfer · approach: Texture and sketch synthesis · 14 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Searches StyleGAN w+ latent space for fixed directions that change lighting (or surface material) while preserving albedo, using intrinsic image decomposition and self-supervised losses, without labeled or CGI data.

Problem

Controlling scene lighting (and resurfacing) in pretrained StyleGAN generated images without labeled attributes, paired data, or CGI supervision, while keeping geometry and albedo fixed.

Approach

Learn a set of latent style-space directions added to w+ such that albedo from an intrinsic decomposition stays consistent while shading varies diversely; directions are jointly found with a classifier that makes them distinguishable, then pruned by greedy forward selection over decomposition-model pools.

Key results

• Forward-selected set of 16 directions yields relighting with cast/soft shadows, inter-reflections, and gloss, with qualitative evaluation only for realism.
• Fine-tuning Omnidata normal predictor on StyLitGAN relights reduces normal variance under lighting, with 7 relights comparable to real Multilum multi-illuminant finetuning (25 relights best).
• Relighted sets have higher FID than vanilla StyleGAN (e.g., faces 5.02 to 37.87, bedroom 5.01 to 14.23), interpreted as out-of-distribution rather than lower quality.

Measured effects reported in the paper

Surprising findings

• GAN-control fails on indoor scenes, changing layout and albedo substantially.
• Higher FID is reported as evidence of out-of-distribution coverage rather than lower realism.

Limitations

• Evaluation of realism is qualitative; FID does not measure realism here.
• Requires careful choice of loss weights and selection among many decomposition models.
• Relighting directions are tied to the pretrained StyleGAN and only shown on specific indoor scene datasets and faces.

Experimental setup

Datasets: LSUN Bedroom Multilum Taskonomy StyleGAN pretrained datasets (bedrooms, faces, churches, conference rooms, kitchens, dining rooms, living rooms)

Benchmarks: Multilum Taskonomy FID on generated sets

Metrics: albedo change (cosine distance) lighting diversity (-log det) clean-FID angular error of surface normals

Baselines: GAN-control Yang et al. relighting directions StyleFlow

Builds on: StyleGAN2 Forsyth and Rock intrinsic image decomposition GAN-control StyleRig

Closest papers by meaning

• LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relighting (CVPR 2025 · similarity 0.88)
• LightIt: Illumination Modeling and Control for Diffusion Models (CVPR 2024 · similarity 0.88)
• Latent Intrinsics Emerge from Training to Relight (NeurIPS 2024 · similarity 0.88)
• ScribbleLight: Single Image Indoor Relighting with Scribbles (CVPR 2025 · similarity 0.87)
• UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting (NeurIPS 2025 · similarity 0.86)
• DreamLight: Towards Harmonious and Consistent Image Relighting (NeurIPS 2025 · similarity 0.86)