Personalized Audio-Driven 3D Facial Animation via Style-Content Disentanglement

Abstract

We present a learning-based approach for generating 3D facial animations with the motion style of a specific subject from arbitrary audio inputs. The subject style is learned from a video clip (1-2 minutes) either downloaded from the Internet or captured through an ordinary camera. Traditional methods often require many hours of the subject's video to learn a robust audio-driven model and are thus unsuitable for this task. Recent research efforts aim to train a model from video collections of a few subjects but ignore the discrimination between the subject style and underlying speech content within facial motions, leading to inaccurate style or articulation. To solve the problem, we propose a novel framework that disentangles subject-specific style and speech content from facial motions. The disentanglement is enabled by two novel training mechanisms. One is two-pass style swapping between two random subjects, and the other is joint training of the decomposition network and audio-to-motion network with a shared decoder. After training, the disentangled style is combined with arbitrary audio inputs to generate stylized audio-driven 3D facial animations. Compared with start-of-the-art methods, our approach achieves better results qualitatively and quantitatively, especially in difficult cases like bilabial plosive and bilabial nasal phonemes.

Supplementary Video

Citation

@article{chai2024personalized,
  author={Chai, Yujin and Shao, Tianjia and Weng, Yanlin and Zhou, Kun},
  journal={IEEE Transactions on Visualization and Computer Graphics},
  title={Personalized audio-driven 3d facial animation via style-content disentanglement},
  year={2024},
  volume={30},
  number={3},
  pages={1803-1820},
  doi={10.1109/TVCG.2022.3230541}
}