Hybrid Deep Learning for Face VerificationYi Sun 1Xiaogang Wang2,3Xiaoou Tang1,31Department of Information Engineering, The Chinese University of Hong Kong2Department of Electronic Engineering, The Chinese University of Hong Kong3Shenzhen Institutes of Advanced Technology, Chinese Academyof Sciencessy011@ie.cuhk.edu.hkxgwang@ee.cuhk.edu.hkxtang@ie.cuhk.edu.hkAbstractThis paper proposes a hybrid convolutional network(ConvNet)-Restricted Boltzmann Machine (RBM) model forface verification in wild conditions.Akey contributionof this work is to directly learn relational visual features,which indicate identity similarities, from raw pixels of facepairs with a hybrid deep network.The deep ConvNetsin our model mimic the primary visual cortex to jointlyextract local relational visual features from two face imagescompared with the learned filter pairs.These relationalfeatures are further processed through multiple layers toextract high-level and global features. Multiple groups ofConvNets are constructed in order to achieve robustnessand characterize face similarities from different aspects.The top-layer RBM performs inference from complementaryhigh-level features extracted from different ConvNet groupswith a two-level average pooling hierarchy.The entirehybrid deep network is jointly fine-tuned to optimize for thetask of face verification. Our model achieves competitiveface verification performance on the LFW dataset.1. IntroductionFace recognition has been extensively studied in recentdecades [29, 28, 30, 1, 16, 5, 33, 12, 6, 3, 7, 25, 34].This paper addresses the key challenge of computing thesimilarity of two face images given their large intra-personal variations in poses, illuminations, expressions,ages, makeups, and occlusions. It becomes more difficultwhen...
