Face It!: A Pipeline for Real-Time Performance-Driven Facial Animation

Jilliam Maria Diaz Barros, Vladislav Golyanik, Kiran Varanasi, Didier Stricker

In: IEEE (editor). 25th. IEEE International Conference on Image Processing (ICIP-2019) September 22-25 Taipei Taiwan ISBN 978-1-5386-6249-6 IEEE 2019.


This paper presents a new lightweight approach for real-time performance-driven facial animation from monocular videos. We transfer facial expressions from 2D images to a 3D virtual character, by estimating the rigid head pose and non-rigid face deformation from detected and tracked 2D facial landmarks. We map the input face into the facial expression space of the 3D head model using blendshape models and formulate a lightweight energy-based optimization problem, which is solved by non-linear least squares at 18 FPS on a single CPU. Our method robustly handles varying head poses and different facial expressions, including moderately asymmetric ones. Compared to related methods, our approach does not require training data, specialised camera setups or graphics cards, and is suitable for embedded systems. We support our claims with several experiments.


ICIP2019.pdf (pdf, 6 MB)

German Research Center for Artificial Intelligence
Deutsches Forschungszentrum für Künstliche Intelligenz