Deep Generative Prior

Slides (pdf)  Size: 9.33MB

Abstract

Learning image prior is essential to solving various tasks of image restoration and manipulation, such as image colorization, image inpainting, and super-resolution. In the past decades, many image priors have been proposed to capture statistics of natural images. Despite their successes, these priors often serve a dedicated purpose. In this talk, I will share our efforts of leveraging Generative Adversarial Networks (GANs) that are trained on large-scale natural images for richer priors. Deep generative prior (DGP) offers compelling results in restoring missing semantics, e.g., color, patch, resolution, in degraded images. It also enables various image manipulation, including random jittering and image morphing. I will further show the possibility of mining 3D geometric clues from an off-the-shelf 2D GAN that is trained on RGB images only. We found that a pre-trained GAN indeed contains rich 3D knowledge and thus can be used to recover 3D shape from a single 2D image in an unsupervised manner. The recovered 3D shapes immediately allow high-quality image editing like relighting and object rotation. Lastly, I will discuss the way to overcome the efficiency bottleneck in GAN inversion so that we can use a pre-trained GAN for efficient and effective large-factor image super-resolution.

Biography

Chen Change Loy is a Nanyang Associate Professor with the School of Computer Science and Engineering, Nanyang Technological University, Singapore. He is also an Adjunct Associate Professor at the Chinese University of Hong Kong. He received his PhD (2010) in Computer Science from the Queen Mary University of London. Prior to joining NTU, he served as a Research Assistant Professor at the MMLab of the Chinese University of Hong Kong, from 2013 to 2018. He was a postdoctoral researcher at Queen Mary University of London and Vision Semantics Limited, from 2010 to 2013.

His research interests include computer vision and deep learning. He serves as an Associate Editor of the International Journal of Computer Vision (IJCV) and IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). He also serves/served as an Area Chair of CVPR 2021, CVPR 2019, ECCV 2018, BMVC 2020, BMVC 2019 and BMVC 2018. He is a senior member of IEEE.

Chen Change Loy
Nanyang Technological University, Singapore