talking_head.md

Talking Head

• Talking Head
  • Survey
  • Talking Head
  • Projects
  • Datasets
  • Toolkits
  • Products
  • Misc

Survey

Talking Head

• JoyGen: Audio-Driven 3D Depth-Aware Talking-Face Video Editing, arXiv, 2501.01798, arxiv, pdf, cication: -1

  Qili Wang, Dajiang Wu, Zihang Xu, ..., Junshi Huang, Jun Lv

• LatentSync: Audio Conditioned Latent Diffusion Models for Lip Sync, arXiv, 2412.09262, arxiv, pdf, cication: -1

  Chunyu Li, Chao Zhang, Weikai Xu, ..., Bingyue Peng, Weiwei Xing · (LatentSync - bytedance)

• Real-time One-Step Diffusion-based Expressive Portrait Videos Generation, arXiv, 2412.13479, arxiv, pdf, cication: -1

  Hanzhong Guo, Hongwei Yi, Daquan Zhou, ..., Michael Lingelbach, Yizhou Yu

• VividFace: A Diffusion-Based Hybrid Framework for High-Fidelity Video Face Swapping, arXiv, 2412.11279, arxiv, pdf, cication: -1

  Hao Shao, Shulun Wang, Yang Zhou, ..., Yu Liu, Hongsheng Li

• VQTalker: Towards Multilingual Talking Avatars through Facial Motion Tokenization, arXiv, 2412.09892, arxiv, pdf, cication: -1

  Tao Liu, Ziyang Ma, Qi Chen, ..., Xie Chen, Kai Yu · (x-lance.github)

• MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation, arXiv, 2412.04448, arxiv, pdf, cication: -1

  Longtao Zheng, Yifan Zhang, Hanzhong Guo, ..., Bo An, Shuicheng Yan · (memoavatar.github)

• INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations, arXiv, 2412.04037, arxiv, pdf, cication: -1

  Yongming Zhu, Longhao Zhang, Zhengkun Rong, ..., Shuang Liang, Zhipeng Ge · (grisoon.github)

• Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Diffusion Transformer Networks, arXiv, 2412.00733, arxiv, pdf, cication: -1

  Jiahao Cui, Hui Li, Yun Zhan, ..., Jingdong Wang, Siyu Zhu · (hallo3 - fudan-generative-vision)

• ShowMaker: Creating High-Fidelity 2D Human Video via Fine-Grained Diffusion Modeling

• Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis, arXiv, 2411.19509, arxiv, pdf, cication: -1

  Tianqi Li, Ruobing Zheng, Minghui Yang, ..., Jingdong Chen, Ming Yang

• FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait, arXiv, 2412.01064, arxiv, pdf, cication: -1

  Taekyung Ki, Dongchan Min, Gyeongsu Chae

  · (deepbrainai-research.github)

• EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animation, arXiv, 2411.10061, arxiv, pdf, cication: -1

  Rang Meng, Xingyu Zhang, Yuming Li, ..., Chenguang Ma · (antgroup.github)

• X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attention, arXiv, 2403.15931, arxiv, pdf, cication: -1

  You Xie, Hongyi Xu, Guoxian Song, ..., Yichun Shi, Linjie Luo · (X-Portrait - bytedance)

• JoyVASA: Portrait and Animal Image Animation with Diffusion-Based Audio-Driven Facial Dynamics and Head Motion Generation, arXiv, 2411.09209, arxiv, pdf, cication: -1

  Xuyang Cao, Guoxin Wang, Sheng Shi, ..., Jintao Fei, Minyu Gao · (JoyVASA - jdh-algo) · (jdh-algo.github)

• Comparative Analysis of Audio Feature Extraction for Real-Time Talking Portrait Synthesis, arXiv, 2411.13209, arxiv, pdf, cication: -1

  Pegah Salehi, Sajad Amouei Sheshkal, Vajira Thambawita, ..., Michael A. Riegler, Pål Halvorsen

• X-Portrait 2: Highly Expressive Portrait Animation

• HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models, arXiv, 2410.22901, arxiv, pdf, cication: -1

  Shengkai Zhang, Nianhong Jiao, Tian Li, ..., Boya Niu, Jun Gao · (HelloMeme - HelloVision) · (songkey.github)

• MuseTalk: Real-Time High Quality Lip Synchronization with Latent Space Inpainting, arXiv, 2410.10122, arxiv, pdf, cication: -1

  Yue Zhang, Minhao Liu, Zhaokang Chen, ..., Junxin Huang, Wenjiang Zhou

• DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generation, arXiv, 2410.13726, arxiv, pdf, cication: -1

  Hanbo Cheng, Limin Lin, Chenyu Liu, ..., Jun Du, Jia Pan · (hanbo-cheng.github) · (DAWN-pytorch - Hanbo-Cheng)

Projects

• DEGSTalk - CVI-SZU

  Decomposed Per-Embedding Gaussian Fields for Hair-Preserving Talking Face Synthesis

• Linly-Talker - Kedreamix

Datasets

Toolkits

Products

• Meetings are a problem of the past 𝕏
• Build immersive AI-generated video experiences in your application

Misc

• a realistic facial animation system for Meta Quest using Stable Diffusion

  · (v.redd)

• LipSync and Face Operations 🤗