Film, the seventh art, is one of the most important art forms. The birth of silent movies a century ago is the legacy of the movie world and human history. With digital image processing techniques and Artificial Intelligence (AI) technology, Dr. Renjie Wan, Assistant Professor of the Department of Computer Science, HKBU, is giving classical silent movies a second life.
Recently approved by the Blue Sky Research Fund, Dr. Wan’s latest research project, “The rejuvenation of classical silent movie using AI”, aims to bring monochrome silent movies back to life. Set up by HKBU, the Blue Sky Research Fund encourages curiosity-driven research work to push the frontiers of science and advance knowledge without the restrictions of a pre-defined agenda.
“Classical silent movies are in monochrome, without any colours. Our first research idea is to think about how to colourise the monochrome image frames using the image processing technique and computer vision, transforming the monochrome counterpart to colourful version which can be enjoyed by the audience nowadays,” says Dr. Wan, whose research interests include computational photography, 3D vision, and AI security.
The project is also trying to use the latest AI technology to enhance the classical silent movie viewing experience. “Audiences nowadays have adapted to audio-video immersive experience. Classical silent movies do not have any audio, we need to create some audio experience of the rejuvenated movies by generating some audios,” he says.
For instance, the project is developing AI algorithm to analyse the oral movement of the actors in the movies and transfer such information into audio.
In classical silent movies, inter-titles are often used to connect different frames to tell the stories. But some important contents and video frames of the silent movies are missing.
“By deploying generative models, some important contents and missing video frames in classical silent movies are generated based on the descriptions in the inter-titles to complement or replace the old frames,” says Dr. Wan, who received his BEng degree in Network Engineering from the University of Electronic Science and Technology and his PhD degree from the Interdisciplinary Graduate School of Nanyang Technological University in Singapore.
“If we try our best to protect the Forbidden City in Beijing or Mona Lisa in Paris or any paintings in the world, we also need to try our best to protect classical silent movies. Silent movies are the most historical digital resources in our current society. They are our cultural heritage which recorded a very important page in our history,” says Dr. Wan.
Dr. Wan’s previous research projects mainly focus on image processing or other image-related issues, such as reflection removal, low-light image enhancement and image restoration. From 2020 to 2021, he was a Wallenberg-NTU Presidential Postdoctoral Fellow supported by Wallenberg Foundation in Sweden and NTU. Dr. Wan has several publications in top journals and conferences (TPAMI, IJCV, TIP, CVPR, and ICCV). He was also the recipient of the outstanding reviewer of ICCV 2019 and the VCIP 2020 Best Paper Award.
His previous research aims to help people and machines see clearly when an image is captured through transparent glass or under low-light conditions. “During the process, we used machine learning technology to make the algorithms more robust. For example, we may use such technology to make our algorithm adapt to different scenarios — two different glasses or lighting conditions,” he says.
When Dr. Wan first began to work on the reflection removal area when he was at Nanyang Technological University about seven years ago, there were only two or three papers existing in the area, and they were all done by famous universities such as MIT and National University of Singapore. “Since then, we did quite a lot of contributions to this area and this area has also become a very important subarea in computational photography due to our contributions,” he says.
When he was a guest researcher at Peking University from 2019 to 2020, he took part in a research project using AI technology to create new Chinese ink art based on some real-world images captured, transferring a picture captured by a mobile phone to Chinese ink art.
Two years ago, he also helped develop a person re-identification system for Singapore’s Covid-19 Foreign Worker Isolation Facilities. Some algorithms were deployed in the surveillance camera to help identify unusual or abnormal situations, and help identify persons that might need help in the facility. “Unexpected things may happen all the times, especially at night when the lighting condition is not good. How to effectively monitor the situations under low-light conditions is what I contributed to this area,” he says.
Came to HKBU earlier this year, Dr. Wan turned his research interest into combining his research experience in computational photography with 3D vision. “For example, when you capture an image in the real world, they are influenced by different artefacts such as reflections and lighting conditions. Such kind of artefacts will influence the quality of the 3D models we build,” he says. “This is an important research project that set my research foundation at HKBU.”
A fan of photography, Dr. Wan is also collaborating with Peking University, to deploy AI algorithms into some newly developed cameras while continuing to conduct more research on AI for arts.
