Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image
CVPR 2024

scales

Holo-Relighting performs volumetric relighting on a single input portrait image, allowing users to individually control (1) lighting effect via an environment map, (2) camera viewpoint and (3) head pose. It is highly expressive and can render complex illumination effects on in-the-wild human faces with accurate view consistency (a). Controls are well disentangled to produce a realistic rendering of moving shadows cast by a point light while rotating the head (b). Practical photographic applications such as shadow diffusion (softening) are also made feasible with our system (c).



Abstract

At the core of portrait photography is the search for ideal lighting and viewpoint. The process often requires advanced knowledge in photography and an elaborate studio setup. In this work, we propose Holo-Relighting, a volumetric relighting method that is capable of synthesizing novel viewpoints, and novel lighting from a single image. Holo-Relighting leverages the pretrained 3D GAN (EG3D) to reconstruct geometry and appearance from an input portrait as a set of 3D-aware features. We design a relighting module conditioned on a given lighting to process these features, and predict a relit 3D representation in the form of a tri-plane, which can render to an arbitrary viewpoint through volume rendering. Besides viewpoint and lighting control, Holo-Relighting also takes the head pose as a condition to enable head-pose-dependent lighting effects. With these novel designs, Holo-Relighting can generate complex non-Lambertian lighting effects (e.g., specular highlights and cast shadows) without using any explicit physical lighting priors. We train Holo-Relighting with data captured with a light stage, and propose two data-rendering techniques to improve the data quality for training the volumetric relighting system. Through quantitative and qualitative experiments, we demonstrate Holo-Relighting can achieve state-of-the-arts relighting quality with better photorealism, 3D consistency and controllability.

Method

scales

An overview of Holo-Relighting. Our method consists of three stages. (a) We first remove the shading from the input portrait and estimate an albedo image. (b) We then conduct GAN inversion upon EG3D to obtain a latent code encoding 3D information of the subject. (c) The relighting network takes in the lighting condition, head pose as well as intermediate GAN features produced by EG3D’s tri-plane generator using the inverted latent code w, and predicts a shading tri-plane, which is summed to the albedo, resulting in the relit tri-plane with lighting embedded. High-resolution RGB images can be rendered from rrelit tri-plane via volume rendering and a super-resolution network. During training, we freeze tri-plane generator and only update the relighting net.

BibTeX 

@article{mei2024holo,
  title={Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image},
  author={Mei, Yiqun and Zeng, Yu and Zhang, He and Shu, Zhixin and Zhang, Xuaner and Bi, Sai and Zhang, Jianming and Jung, HyunJoon and Patel, Vishal M},
  journal={arXiv preprint arXiv:2403.09632},
  year={2024}
}
The website template was borrowed from Nerfies.