HONG KONG BAPTIST UNIVERSITY
FACULTY OF SCIENCE
Department of Computer Science Seminar
Towards Robust Deep Learning
Mr. Bo Han
Center for Advanced Intelligence Project
Date: December 20, 2018 (Thursday)
Time: 4:30 - 5:30 pm
Venue: SCT909, Cha Chi Ming Science Tower, Ho Sin Hang Campus
It is challenging to train deep neural networks robustly with noisy labels, as the capacity of deep neural networks is so high that they can totally overfit on these noisy labels. In this talk, I will introduce three orthogonal techniques in robust deep learning with noisy labels, namely data perspective “estimating the noise transition matrix”; training perspective “training on selected samples”; and regularization perspective “conducting scaled stochastic gradient ascent”. First, as an approximation of real-world corruption, noisy labels are corrupted from ground-truth labels by an unknown noise transition matrix. Thus, the accuracy of classifiers can be improved by estimating this matrix. We present a human-assisted approach called “Masking”. Masking conveys human cognition of invalid class transitions, and naturally speculates the structure of the noise transition matrix. Given the structure information, we only learn the noise transition probability to reduce the estimation burden. Second, motivated by the memorization effects of deep networks, which shows networks fit clean instances first and then noisy ones, we present a new paradigm called “Co-teaching” even combating with extremely noisy labels. We train two networks simultaneously. First, in each mini-batch data, each network filters noisy instances based on the memorization effects. Then, it teaches the remaining instances to its peer network for updating the parameters. Third, deep networks inevitably memorize some noisy labels, which will degrade their generalization. We propose a meta algorithm called “Pumpout” to overcome the problem of memorizing noisy labels. By using scaled stochastic gradient ascent, Pumpout actively squeezes out the negative effects of noisy labels from the training model, instead of passively forgetting these effects. We leverage Pumpout to upgrade two representative methods: MentorNet and Backward Correction.
Bo Han is a final-year Ph.D. student at Centre for Artificial Intelligence, University of Technology Sydney, advised by Prof. Ivor W. Tsang. He is currently a research intern at Center for Advanced Intelligence Project, RIKEN, working on Robust Deep Learning projects with Prof. Masashi Sugiyama and Dr. Gang Niu. His current research interests lie in machine learning and its real-world applications. His long-term goal is to develop intelligent systems, which can learn from a massive volume of complex (uncertain, adversarial, private, and interactive) data. He has published 10 journal articles and conference papers, including 3 MLJ articles and 2 NeurIPS papers.
********* ALL INTERESTED ARE WELCOME ***********
(For enquiry, please contact Computer Science Department at 3411 2385)
Department of Computer Science, Hong Kong Baptist University