A curated list of sign language procesing (e.g., sign language recognition, sign language translation) and related area (e.g., speech translation, motion generation) resources. The old version can be found here.
- [Wiki] Sign Language
- [Wiki] Sign Language Recognition
- [Wiki] Gesture_recognition
- [Wiki] Sign Language Tranlsation
- [Resource] jinwchoi/awesome-action-recognition
- [Resource] xinghaochen/awesome-hand-pose-estimation
- [Resource] OpenHuman-ai/awesome-gesture_generation
- [Resource] zzw922cn/awesome-speech-recognition-speech-synthesis-papers
- [CVPR 2024] SignGraph: A Sign Sequence is Worth Graphs of Nodes. [paper] [code]
Shiwei Gan, Yafeng Yin, Zhiwei Jiang, Hongkai Wen, Kang Xia, Lei Xie, Sanglu Lu - [EMNLP Findings 2023] Handshape-Aware Sign Language Recognition: Extended Datasets and Exploration of Handshape-Inclusive Methods. [paper]
Xuan Zhang, Kevin Duh - [EMNLP Findings 2023] Making Body Movement in Sign Language Corpus Accessible for Linguists and Machines with Three-Dimensional Normalization of MediaPipe. [paper]
Houda Bouamor, Juan Pino, Kalika Bali - [EMNLP Findings 2023] Linguistically Motivated Sign Language Segmentation. [paper]
Amit Moryossef, Zifan Jiang, Mathias Müller, Sarah Ebling, Yoav Goldberg - [NeurIPS 2023 Dataset] ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognition. [paper]
Aashaka Desai, Lauren Berger, Fyodor O Minakov, Vanessa Milan, Chinmay Singh, Kriston L Pumphrey, Richard Ladner, Hal Daumé III, Alex Xijie Lu, Naomi Caselli, Danielle Bragg - [NeurIPS 2023 Dataset] PopSign ASL v1.0: An Isolated American Sign Language Dataset Collected via Smartphones. [paper]
Thad Starner, Sean Forbes, Matthew So, David Martin, Rohit Sridhar, Gururaj Deshpande, Sam Sepah, Sahir Shahryar, Khushi Bhardwaj, Tyler Kwok, Daksh Sehgal, Saad Hassan, Bill Neubauer, Sofia Anandi Vempala, Alec Tan, Jocelyn Heath, Unnathi Utpal Kumar, Priyanka Vijayaraghavan Mosur, Tavenner M. Hall, Rajandeep Singh et al. - [ICCV 2023] CoSign: Exploring Co-occurrence Signals in Skeleton-based Continuous Sign Language Recognition. [paper]
Peiqi Jiao, Yuecong Min, Yanan Li, Xiaotao Wang, Lei Lei, Xilin Chen - [ICCV 2023] Human Part-wise 3D Motion Context Learning for Sign Language Recognition. [paper]
Taeryung Lee, Yeonguk Oh and Kyoung Mu Lee - [ICCV 2023] Improving Continuous Sign Language Recognition with Cross-Lingual Signs. [paper]
Fangyun Wei, Yutong Chen - [ICCV 2023] C2ST: Cross-modal Contextualized Sequence Transduction for Continuous Sign Language Recognition. [paper]
Huaiwen Zhang, Zihang Guo, Yang Yang, Xin Liu, De Hu - Complete List of Sign Language Recognition
- [ECCV 2024] Visual Alignment Pre-training for Sign Language Translation. [paper]
Peiqi Jiao, Yuecong Min, Xilin Chen - [ECCV 2024] EvSign: Sign Language Recognition and Translation with Streaming Events. [paper]
Pengyu Zhang, Hao Yin, Zeren Wang, Wenyue Chen, Sheng Ming Li, Dong Wang, Huchuan Lu, Xu Jia - [ACL 2024] Towards Privacy-Aware Sign Language Translation at Scale. [paper]
Phillip Rust, Bowen Shi, Skyler Wang, Necati Cihan Camgoz, Jean Maillard - [ACL 2024] Sign Language Translation with Sentence Embedding Supervision. [paper]
HAMIDULLAH Yasser, Josef van Genabith, Cristina España-Bonet - [ACL Findings 2024] iSign: A Benchmark for Indian Sign Language Processing. [paper]
Abhinav Joshi, Romit Mohanty, Mounika Kanakanti, Andesha Mangla, Sudeep Choudhary, Monali Barbate, Ashutosh Modi - [ACL Findings 2024] Unsupervised Sign Language Translation and Generation. [paper]
Zhengsheng Guo, Zhiwei He, Wenxiang Jiao, Xing Wang, Rui Wang, Kehai Chen, Zhaopeng Tu, Yong Xu, Min Zhang - [CVPR 2024] LLMs are Good Sign Language Translators. [paper]
Jia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani, Jun Liu - [ICLR 2024] Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation. [paper] Ryan Wong, Necati Cihan Camgoz, Richard Bowden
- [EMNLP Findings 2023] JWSign: A Highly Multilingual Corpus of Bible Translations for more Diversity in Sign Language Processing. [paper]
Shester Gueuwou, Sophie Siake, Colin Leong, Mathias Müller - [EMNLP Findings 2023] Cross-modality Data Augmentation for End-to-End Sign Language Translation. [paper]
Jinhui Ye, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, Hui Xiong - [OpenReview] Towards Faithful Sign Language Translation. [paper]
Yuecong Min, Xilin Chen - [NeurIPS 2023 Dataset] YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus. [paper]
David Uthus, Garrett Tanzer, Manfred Georg - [NeurIPS 2023 Dataset] Auslan-Daily: Australian Sign Language Translation for Daily Communication and News. [paper]
Xin Shen, Shaozu Yuan, Hongwei Sheng, Heming Du, Xin Yu - [ICCV 2023] Sign Language Translation with Iterative Prototype. [paper]
Huijie Yao, Wengang Zhou, Hao Feng, Hezhen Hu, Hao Zhou, Houqiang Li - [ICCV 2023] Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining. [paper]
Benjia Zhou, Zhigang Chen, Albert Clapés, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, Du Zhang - Complete List of Sign Language Translation
- [ECCV 2024] A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars. [paper]
Ronglai Zuo, Fangyun Wei, Zenggui Chen, Brian Mak, Jiaolong Yang, Xin Tong - [ECCV 2024] Pose Guided Fine-Grained Sign Language Video Generation. [paper]
Tongkai Shi, Lianyu Hu, Fanhua Shang, Jichao Feng, liu peidong, Wei Feng - [ECCV 2024] SignGen: End-to-End Sign Language Video Generation with Latent Diffusion. [paper]
Fan Qi, Yu Duan, Changsheng Xu, Huaiwen Zhang - [ACL 2024] T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text. [paper]
Aoxiong Yin, Haoyuan Li, Kai Shen, Siliang Tang, Yueting Zhuang - [ACL Findings 2024] MS2SL: Multimodal Spoken Data-Driven Continuous Sign Language Production. [paper]
Jian Ma, Wenguan Wang, Yi Yang, Feng Zheng - [CVPR 2024] Neural Sign Actors: A diffusion model for 3D sign language production from text. [paper]
Vasileios Baltatzis, Rolandos Alexandros Potamias, Evangelos Ververas, Guanxiong Sun, Jiankang Deng, Stefanos Zafeiriou - [WACV 2024] Sign Language Production With Latent Motion Transformer. [paper]
Pan Xie, Taiying Peng, Yao Du, Qipeng Zhang - [CVPR 2022] Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Production. [paper]
Ben Saunders, Necati Cihan Camgoz, Richard Bowden - [ICCV 2021] Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives. [paper]
Ben Saunders; Necati Cihan Camgoz; Richard Bowden - Complete List of Sign Language Production
- [ECCV 2024] Uncertainty-aware sign language video retrieval with probability distribution modeling. [paper]
Xuan Wu, Hongxiang Li, yuanjiang luo, Xuxin Cheng, Xianwei Zhuang, Meng Cao, Keren Fu - [ECCV 2024] SignAvatars: A Large-scale 3D Sign Language Holistic Motion Dataset and Benchmark. [paper]
Zhengdi Yu, Shaoli Huang, yongkang cheng, Tolga Birdal - [ACL 2024] Semi-Supervised Spoken Language Glossification. [paper]
Huijie Yao, Wengang Zhou, Hao Zhou, Houqiang Li - [ACL 2024] American Sign Language Handshapes Reflect Pressures for Communicative Efficiency. [paper]
Kayo Yin, Terry Regier, Dan Klein - [EMNLP Findings 2023] Enhancing Accessible Communication: from European Portuguese to Portuguese Sign Language. [paper]
Catarina Sousa, Luisa Coheur, Mara Moita - [CVPR 2023] CiCo: Domain-Aware Sign Language Retrieval via Cross-Lingual Contrastive Learning. [paper][code]
Yiting Cheng, Fangyun Wei, Jianmin Bao, Dong Chen, Wenqiang Zhang - [CVPR 2023] Ham2Pose: Animating Sign Language Notation into Pose Sequences. [paper]
Rotem Shalev Arkushin, Amit Moryossef, Ohad Fried
- [ECCV 2024] Co-speech Gesture Video Generation with 3D Human Meshes Gestures [paper]
Aniruddha Mahapatra, Richa Mishra, Ziyi Chen, Boyang Ding, Renda Li, Shoulei Wang, Jun-Yan Zhu, Peng Chang, Mei Han, Jing Xiao - [ACL 2024] LLM Knows Body Language, Too: Translating Speech Voices into Human Gestures. [paper]
Chenghao Xu, Guangtao Lyu, Jiexi Yan, Muli Yang, Cheng Deng - [CVPR 2024] EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling. [paper]
Haiyang Liu, Zihao Zhu, Giorgio Becherini, YICHEN PENG, Mingyang Su, YOU ZHOU, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, Michael J. Black - [CVPR 2024] Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generation. [paper]
Xingqun Qi, Jiahao Pan, Peng Li, Ruibin Yuan, Xiaowei Chi, Mengfei Li, Wenhan Luo, Wei Xue, Shanghang Zhang, Qifeng Liu, Yike Guo - [CVPR 2024] Towards Variable and Coordinated Holistic Co-Speech Motion Generation. [paper] Yifei Liu, Qiong Cao, Yandong Wen, Huaiguang Jiang, Changxing Ding
- [CVPR 2024] ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis. [paper] Muhammad Hamza Mughal, Rishabh Dabral, Ikhsanul Habibie, Lucia Donatelli, Marc Habermann, Christian Theobalt
- [CVPR 2024] Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model. [paper] Xu He, Qiaochu Huang, Zhensong Zhang, Zhiwei Lin, Zhiyong Wu, Sicheng Yang, Minglei Li, Zhiyi Chen, Songcen Xu, Xiaofei Wu
- [ICCV 2023] Continual Learning for Personalized Co-speech Gesture Generation. [paper]
Chaitanya Ahuja, Pratik Joshi, Ryo Ishii, Louis-Philippe Morency - [ICCV 2023] LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation. [paper]
Yihao Zhi, Xiaodong Cun, Xuelin Chen, Xi Shen, Wen Guo, Shaoli Huang, Shenghua Gao - [CVPR 2023] QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation. [paper]
SiCheng Yang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, Haolin Zhuang - [CVPR 2023] Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation. [paper]
Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, Lequan Yu - [CVPR 2023] Diverse 3D Hand Gesture Prediction from Body Dynamics by Bilateral Hand Disentanglement. [paper]
Xingqun Qi, Chen Liu, Muyi Sun, Lincheng Li, Changjie Fan, Xin Yu - [CVPR 2023] Co-speech Gesture Synthesis by Reinforcement Learning with Contrastive Pre-trained Rewards. [paper]
Mingyang Sun, Mengchen Zhao, Yaqing Hou, Minglei Li, Huang Xu, Songcen Xu, Jianye HAO - Complete List of Co-speech Gesture Generation
- [ICCV 2023] Learning Robust Representations with Information Bottleneck and Memory Network for RGB-D-based Gesture Recognition. [paper]
Yunan Li, Huizhou Chen, Guanwen Feng, Qiguang Miao - [ICCV 2023] Data-Free Class-Incremental Hand Gesture Recognition. [paper]
Shubhra Aich, Jesus Ruiz-Santaquiteria, Zhenyu Lu, Prachi Garg, K J Joseph, Alvaro Fernandez Garcia, Vineeth N Balasubramanian, Kenrick Kin, Chengde Wan, Necati Cihan Camgoz, Shugao Ma, Fernando De la Torre - [BMVC 2022] Towards more efficient few-shot learning based human gesture recognition via dynamic vision sensors. [paper]
Linglin Jing, Yifan Wang, Tailin Chen, Shirin Dora, Zhigang Ji, Hui Fang - [BMVC 2022] Continuous Hand Gesture Recognition using Deep Coarse
and Fine Hand Features. [paper]
Hazem Wannous, Jean-Philippe Vandeborre - [CVPR 2022] LD-ConGR: A Large RGB-D Video Dataset for Long-Distance Continuous Gesture Recognition. [paper]
Dan Liu; Libo Zhang; Yanjun Wu - [AAAI 2022] Learning Unseen Emotions from Gestures via Semantically-Conditioned Zero-Shot Perception with Adversarial Autoencoders. [paper]
Abhishek Banerjee, Uttaran Bhattacharya, Aniket Bera - Complete List of Gesture Recognition
- [ACL LoResMT 2024] The Seventh Workshop on Technologies for Machine Translation of Low-Resource Languages
- [ICCV ACVR 2023] Eleventh International Workshop on Assistive Computer Vision and Robotics
- [ECCV SLRTP 2020] Sign language recognition, translation & production. [Accepted papers]
| Dataset | Task | Modality | Statistic | |||||
|---|---|---|---|---|---|---|---|---|
| Name | Language | Isolated | Continuous | RGB | Depth | #Signers | #Vocab | #Sequences |
| DGS Kinect | DGS | √ | √ | √ | 15 | 40 | 3,000 | |
| ASLLVD | ASL | √ | √ | 6 | 2,742 | 9,794 | ||
| DEVISIGN-L | CSL | √ | √ | √ | 8 | 2,000 | 24,000 | |
| CSL-500 | CSL | √ | √ | 50 | 500 | 125,000 | ||
| MSASL | ASL | √ | √ | 222 | 1,000 | 25,513 | ||
| WLASL2000 | ASL | √ | √ | 119 | 2,000 | 21,083 | ||
| AUTSL | TSL | √ | √ | √ | 43 | 226 | 38,336 | |
| SIGNUM | DGS | √ | √ | √ | 9 | 455 | 2,340 | |
| Phoenix14 | DGS | √ | √ | 9 | 1,200 | 6,841 | ||
| KETI | KSL | √ | √ | 14 | 524 | 15,672 | ||
| GSL | GSL | √ | √ | √ | 7 | 310 | 10,290 | |
| CSL-100 | CSL | √ | √ | 50 | 2,000 | 25,000 | ||
| CSL-Daily | CSL | √ | √ | 10 | 2,000 | 20,654 | ||
| BSL-1K | BSL | √ | √ | 40 | 1,064 | 273K | ||
| VRT-NEWS | VGT | √ | √ | 9 | 6,325 | 7,174 | ||
| How2Sign | BSL | √ | √ | √ | 11 | 15,686 | 35,191 | |
| BOBSL | BSL | √ | √ | 39 | 2,281 | 452K | ||
| OpenASL | ASL | √ | √ | 220 | 33,549 | 98,417 | ||
| YouTube-ASL | ASL | √ | √ | >2519 | 60K | 11,093 |
