MemoryTalker: Personalized Speech-Driven 3D Facial Animation via Audio-Guided Stylization

Hyung Kyu Kim1, Sangmin Lee2,*, Hak Gu Kim3,*

1Department of Imaging Science and Arts, Chung-Ang University, South Korea
2Department of Computer Science and Engineering, Korea University, South Korea
3Department of Metaverse Convergence, Chung-Ang University, South Korea
*Corresponding Author
Paper Supplementary Code arXiv

MemoryTalker: Realistic 3D facial animation from audio alone — no priors, just your voice.

Abstract

The intuition of MemoryTalker for personalized speech-driven 3D facial animation.

Speech-driven 3D facial animation aims to synthesize realistic facial motion sequences from given audio, matching the speaker's speaking style. However, previous works often require priors such as class labels of a speaker or additional 3D facial meshes at inference, which makes them fail to reflect the speaking style and limits their practical use. To address these issues, we propose MemoryTalker which enables realistic and accurate 3D facial motion synthesis by reflecting speaking style only with audio input to maximize usability in applications. Our framework consists of two training stages: The 1-stage is storing and retrieving general motion (i.e., Memorizing), and the 2-stage is to perform the personalized facial motion synthesis (i.e., Animating) with the motion memory stylized by the audio-driven speaking style feature. In this second stage, our model learns about which facial motion types should be emphasized for a particular piece of audio. As a result, our MemoryTalker can generate a reliable personalized facial animation without additional prior information. With quantitative and qualitative evaluations, as well as user study, we show the effectiveness of our model and its performance enhancement for personalized facial animation over state-of-the-art methods.

Poster

BibTeX


        @article{MemoryTalker,
        author = {Kim, Hyung Kyu and Lee, Sangmin and Kim, Hak Gu}, 
        title = {MemoryTalker: Personalized Speech-Driven 3D Facial Animation via Audio-Guided Stylization},
        journal = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, 
        month = {October},
        year = {2025},
        pages = {n-n+7}
        }