Controlling/Editing Object Appearance in Neural Rendering
Dimitris Samaras
Jueves 22 Diciembre 9:00
In this talk I will be presenting recent work on controlling/editing object appearance during neural rendering. Recent work on differentiable rendering techniques for implicit surfaces has shown high quality 3D scene reconstruction and view synthesis results. However, these methods typically learn the appearance color as a function of the surface points and lack explicit surface parameterization. Thus they do not allow texture map extraction or texture editing. We propose an efficient method to learn surface parameterization by learning a continuous bijective mapping between 3D surface positions and 2D texture-space coordinates. Our surface parameterization network can be conveniently plugged into a differentiable rendering pipeline and trained using multiview images and rendering loss. Using the learned parameterized implicit 3D surface we demonstrate state-of-the-art document-unwarping via texture extraction in both synthetic and real scenarios. We also show that our approach can reconstruct and edit high-frequency textures for arbitrary objects. In the second part of the talk I will be talking about Neural Radiance Fields (NeRFs) for dynamic scenes with explicit controls over deforming foreground objects, such as a human head, within a scene. Training such models requires photometric consistency over the deformed region e.g. the face must be evenly lit as it deforms with changing head-pose and facial expression. Such photometric consistency across frames of a video is hard to maintain, even in studio environments, thus making the creation of a reanimatable neural portrait untenable. In this work, we propose CoDyNeRF, a system that enables the creation of fully controllable 3D portraits in real-world capture conditions. CoDyNeRF models changes in lighting via a dynamic appearance model that is conditioned on predicted surface normals and the facial expression and head-pose deformation. The dynamic surface normals prediction is guided using 3DMM normals that act as a coarse prior on the normals of the human head, where direct prediction of normals is hard due to rigid and non-rigid deformations induced by head-pose and facial expression changes .Using only a smartphone-captured short video of a subject for training, our method allows synthesis of a portrait scene with explicit head pose and expression controls and realistic lighting effects.
Bio: Dimitris Samaras received the diploma degree in computer science and engineering from the University of Patras, in 1992, the MSc degree from Northeastern University, in 1994, and the PhD degree from the University of Pennsylvania, in 2001. He is a SUNY Empire Innovation Professor of Computer Science with Stony Brook University, where he directs the Computer Vision Lab. His research interests include human behavior analysis, generative models, illumination modeling and estimation for recognition and graphics, and biomedical image analysis. He has co-authored over 150 peer-reviewed research articles in Computer Vision, Machine Learning, Computer Graphics and Medical Imaging conferences and journals. He was Program Chair of CVPR 2022 and is frequent Area Chair in Computer Vision and Machine Learning conferences.
