-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.xml
559 lines (514 loc) · 36.9 KB
/
index.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<title>Visual Sign Language Research Group</title>
<link>https://ustc-slr.github.io/</link>
<atom:link href="https://ustc-slr.github.io/index.xml" rel="self" type="application/rss+xml" />
<description>Visual Sign Language Research Group</description>
<generator>Wowchemy (https://wowchemy.com)</generator><language>en-us</language><lastBuildDate>Wed, 26 Apr 2023 00:00:00 +0000</lastBuildDate>
<image>
<url>https://ustc-slr.github.io/media/icon_hu393e4df0623b7154086f58675ba04221_25427_512x512_fill_lanczos_center_3.png</url>
<title>Visual Sign Language Research Group</title>
<link>https://ustc-slr.github.io/</link>
</image>
<item>
<title>SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding</title>
<link>https://ustc-slr.github.io/highlight_news/23-05-02-signbert+/</link>
<pubDate>Wed, 26 Apr 2023 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/highlight_news/23-05-02-signbert+/</guid>
<description><p><em>IEEE TPAMI 2023</em></p>
<p>Hand gesture serves as a crucial role during the expression of sign language. Current deep learning based methods for sign language understanding (SLU) are prone to over-fitting due to insufficient sign data resource and suffer limited interpretability.
In this paper, we propose the <em>first</em> self-supervised pre-trainable SignBERT+ framework with model-aware hand prior incorporated. In our framework, the hand pose is regarded as a visual token, which is derived from an off-the-shelf detector. Each visual token is embedded
with gesture state and spatial-temporal position encoding. To take full advantage of current sign data resource, we first perform self-supervised learning to model its statistics. To this end, we design multi-level masked modeling strategies (joint, frame and clip) to
mimic common failure detection cases. Jointly with these masked modeling strategies, we incorporate model-aware hand prior to better capture hierarchical context over the sequence. After the pre-training, we carefully design simple yet effective prediction heads for downstream tasks.
To validate the effectiveness of our framework, we perform extensive experiments on three main SLU tasks,
involving isolated and continuous sign language recognition (SLR), and sign language translation (SLT). Experimental results
demonstrate the effectiveness of our method, achieving new state-of-the-art performance with a notable gain.</p>
<div class="alert alert-note">
<div>
Please refer to this <a href="https://signbert-zoo.github.io/" target="_blank" rel="noopener">project page</a> for more details.
</div>
</div></description>
</item>
<item>
<title>SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding</title>
<link>https://ustc-slr.github.io/publication/hu2023signbert/</link>
<pubDate>Wed, 26 Apr 2023 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/hu2023signbert/</guid>
<description></description>
</item>
<item>
<title>Prior-aware Cross Modality Augmentation Learning for Continuous Sign Language Recognition</title>
<link>https://ustc-slr.github.io/publication/hu2023prior/</link>
<pubDate>Wed, 19 Apr 2023 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/hu2023prior/</guid>
<description></description>
</item>
<item>
<title>BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenization</title>
<link>https://ustc-slr.github.io/highlight_news/23-02-01-best/</link>
<pubDate>Wed, 01 Feb 2023 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/highlight_news/23-02-01-best/</guid>
<description><p><em>AAAI 2023, Oral</em></p>
<p>In this work, we are dedicated to leveraging the BERT pretraining success and modeling the domain-specific statistics
to fertilize the sign language recognition (SLR) model. Considering the dominance of hand and body in sign language
expression, we organize them as pose triplet units and feed
them into the Transformer backbone in a frame-wise manner. Pre-training is performed via reconstructing the masked
triplet unit from the corrupted input sequence, which learns
the hierarchical correlation context cues among internal and
external triplet units. Notably, different from the highly semantic word token in BERT, the pose unit is a low-level signal originally located in continuous space, which prevents the
direct adoption of the BERT cross-entropy objective. To this
end, we bridge this semantic gap via coupling tokenization of
the triplet unit. It adaptively extracts the discrete pseudo label from the pose triplet unit, which represents the semantic
gesture/body state. After pre-training, we fine-tune the pretrained encoder on the downstream SLR task, jointly with
the newly added task-specific layer. Extensive experiments
are conducted to validate the effectiveness of our proposed
method, achieving new state-of-the-art performance on all
four benchmarks with a notable gain.</p>
<div class="alert alert-note">
<div>
Please refer to this <a href="https://ustc-slr.github.io/" target="_blank" rel="noopener">project page</a> for more details.
</div>
</div></description>
</item>
<item>
<title>BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenization</title>
<link>https://ustc-slr.github.io/publication/zhao2023best/</link>
<pubDate>Wed, 01 Feb 2023 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/zhao2023best/</guid>
<description></description>
</item>
<item>
<title>Collaborative Multilingual Continuous Sign Language Recognition: A Unified Framework</title>
<link>https://ustc-slr.github.io/highlight_news/23-05-02-coslr/</link>
<pubDate>Sat, 26 Nov 2022 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/highlight_news/23-05-02-coslr/</guid>
<description><p><em>IEEE TMM 2022</em></p>
<p>Current continuous sign language recognition systems generally target on a single language. When it comes to the multilingual problem, existing solutions often build separate models based on the same network and then train them with their corresponding sign language corpora. Observing that different sign languages share some low-level visual patterns, we argue that it is beneficial to optimize the recognition model in a collaborative way. With this motivation, we propose the first unified framework for multilingual continuous sign language recognition. Our framework consists of a shared visual encoder for visual information encoding, multiple language-dependent sequential modules for long-range temporal dependency learning aimed at different languages, and a universal sequential module to learn the commonality of all languages. An additional language embedding is introduced to distinguish different languages within the shared temporal encoders. Further, we present a max-probability decoding method to obtain the alignment between sign videos and sign words for visual encoder refinement. We evaluate our approach on three continuous sign language recognition benchmarks, i.e., RWTH-PHOENIX-Weather, CSL and GSL-SD. The experimental results reveal that our method outperforms the individually trained recognition models. Our method also demonstrates better performance compared with state-of-the-art algorithms.</p>
<div class="alert alert-note">
<div>
Please refer to this <a href="https://ustc-slr.github.io/" target="_blank" rel="noopener">project page</a> for more details.
</div>
</div></description>
</item>
<item>
<title>Collaborative Multilingual Continuous Sign Language Recognition: A Unified Framework</title>
<link>https://ustc-slr.github.io/publication/hu2022collaborative/</link>
<pubDate>Fri, 18 Nov 2022 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/hu2022collaborative/</guid>
<description></description>
</item>
<item>
<title>Hand-Object Interaction Image Generation</title>
<link>https://ustc-slr.github.io/publication/hu2022handobject/</link>
<pubDate>Tue, 01 Nov 2022 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/hu2022handobject/</guid>
<description></description>
</item>
<item>
<title>Highlights</title>
<link>https://ustc-slr.github.io/highlights/</link>
<pubDate>Mon, 24 Oct 2022 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/highlights/</guid>
<description></description>
</item>
<item>
<title>SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition</title>
<link>https://ustc-slr.github.io/publication/hu2021signbert/</link>
<pubDate>Fri, 01 Oct 2021 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/hu2021signbert/</guid>
<description></description>
</item>
<item>
<title>Global-Local Enhancement Network for NMF-Aware Sign Language Recognition</title>
<link>https://ustc-slr.github.io/publication/hu2021global/</link>
<pubDate>Thu, 22 Jul 2021 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/hu2021global/</guid>
<description></description>
</item>
<item>
<title>Improving Sign Language Translation with Monolingual Data by Sign Back-Translation</title>
<link>https://ustc-slr.github.io/highlight_news/21-02-01-signbt/</link>
<pubDate>Wed, 16 Jun 2021 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/highlight_news/21-02-01-signbt/</guid>
<description><p><em>CVPR 2021</em></p>
<p>Despite existing pioneering works on sign language translation (SLT), there is a non-trivial obstacle, i.e., the limited quantity of parallel sign-text data. To tackle this
parallel data bottleneck, we propose a sign back-translation (SignBT) approach, which incorporates massive spoken
language texts into SLT training. With a text-to-gloss translation model, we first back-translate the monolingual text to
its gloss sequence. Then, the paired sign sequence is generated by splicing pieces from an estimated gloss-to-sign bank at the feature level. Finally, the synthetic parallel data
serves as a strong supplement for the end-to-end training of the encoder-decoder SLT framework. To promote the SLT research, we further contribute CSLDaily, a large-scale continuous SLT dataset. It provides
both spoken language translations and gloss-level annotations. The topic revolves around people&rsquo;s daily lives (e.g.,
travel, shopping, medical care), the most likely SLT application scenario. Extensive experimental results and analysis
of SLT methods are reported on CSL-Daily. With the proposed sign back-translation method, we obtain a substantial
improvement over previous state-of-the-art SLT methods.</p>
<div class="alert alert-note">
<div>
Please refer to this <a href="https://ustc-slr.github.io/" target="_blank" rel="noopener">project page</a> for more details.
</div>
</div></description>
</item>
<item>
<title>Conditional Sentence Generation and Cross-Modal Reranking for Sign Language Translation</title>
<link>https://ustc-slr.github.io/publication/zhao2021conditional/</link>
<pubDate>Mon, 07 Jun 2021 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/zhao2021conditional/</guid>
<description></description>
</item>
<item>
<title>Spatial-Temporal Multi-Cue Network for Sign Language Recognition and Translation</title>
<link>https://ustc-slr.github.io/publication/zhou2021spatial/</link>
<pubDate>Mon, 15 Feb 2021 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/zhou2021spatial/</guid>
<description></description>
</item>
<item>
<title>Improving sign language translation with monolingual data by sign back-translation</title>
<link>https://ustc-slr.github.io/publication/zhou2021improving/</link>
<pubDate>Mon, 01 Feb 2021 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/zhou2021improving/</guid>
<description></description>
</item>
<item>
<title>Model-Aware Gesture-to-Gesture Translation</title>
<link>https://ustc-slr.github.io/publication/hu2021model/</link>
<pubDate>Mon, 01 Feb 2021 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/hu2021model/</guid>
<description></description>
</item>
<item>
<title>CSL-Daily Dataset</title>
<link>https://ustc-slr.github.io/datasets/2021_csl_daily/</link>
<pubDate>Fri, 01 Jan 2021 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/datasets/2021_csl_daily/</guid>
<description><h2>Introduction</h2>
<p class="textBlock">
CSL-Daily is a large-scale continuous SLT dataset.
It provides both spoken language translations and gloss-level annotations.
The topic revolves around people's daily lives (e.g., travel, shopping, medical care), the most likely SLT application scenario.
<!-- We have two Chinese sign language datasets for isolated Sign Language Recognition and continuous Sign Language Recognition, respectively. Both datasets are collected with Kinect 2.0 by 50 signers. Each signer perfors 5 times for every word (sentence). The sign videos are recorded with 30<i>fps</i>. The distance between the signers and Kinect is about 1.5 meters. Each instance in both datasets contains <b>RGB videos</b>, <b>depth videos</b>, and <b>3D joints information</b> of the signer. -->
</p>
<h2>Download </h2>
<p class="textBlock">
The CSL-Daily database is released to universities and research institutes for research purpose only. To request the access right to the data resources, please follow the instructions below:
</p><ol>
<li>Download the <a href="./Release-Agreement-CSL-Daily.pdf" target="_blank"><font color="#FF0000">CSL-Daily Dataset Release Agreement</font></a>;
</li><li><font color="#FF0000">Read all items and conditions carefully</font>;
</li><li>Complete it appropriately. Note that the agreement should be signed by <font color="#FF0000">a full-time staff member</font> (that is, the student is not acceptable).
</li><li>Please scan the signed agreement, send it to (ustc_vslrg At 126.com) and CC to Prof. Zhou (zhwg At ustc.edu.cn). If you are a student, please also CC to the full-time staff member who sign the agreement.
</li></ol>
<p></p>
<h2>Reference</h2>
<p class="textBlock">
Please cite the following papers if you use CSL-Daily for your research
</p><ul>
<li>Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li, "Improving Sign Language Translation with Monolingual Data by Sign Back-Translation," <i>IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR)</i>, 2021.</li>
</ul>
<p></p>
<p class="textBlock">
Besides, you can refer to the following papers for continuous SLR published by our group:
</p><ul>
<li>Hezhen Hu, Weichao Zhao, Wengang Zhou, and Houqiang Li, "SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding," <i>IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI)</i>, 2023.</li>
</ul>
<p></p>
<h2>Contact</h2>
<p class="textBlock">
If you have any questions about the dataset and our papers, please feel free to contact us:
</p><ul>
<li><a href="http://staff.ustc.edu.cn/~zhwg" target="_blank">Wengang Zhou</a>, Professor, USTC, zhwg AT ustc.edu.cn</li>
<li><a href="http://staff.ustc.edu.cn/~lihq/English.html" target="_blank">Houqiang Li</a>, Professor, USTC, lihq AT ustc.edu.cn</li>
<li><a href="https://ustc-slr.github.io/" target="_blank">Weichao Zhao</a>, Ph.D Student, USTC, saruka AT mail.ustc.edu.cn</li>
</ul>
<p></p>
<br></description>
</item>
<item>
<title>Boosting Continuous Sign Language Recognition via Cross Modality Augmentation</title>
<link>https://ustc-slr.github.io/highlight_news/20-11-10-cma/</link>
<pubDate>Fri, 20 Nov 2020 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/highlight_news/20-11-10-cma/</guid>
<description><p><em>ACM MM 2020</em></p>
<p>Continuous sign language recognition (SLR) deals with unaligned video-text pair and uses the word error rate (WER), i.e., edit distance, as the main evaluation metric. Since it is not differentiable, we usually instead optimize the learning model with the connectionist temporal classification (CTC) objective loss, which maximizes the posterior probability over the sequential alignment. Due to the optimization gap, the predicted sentence with the highest decoding probability may not be the best choice under the WER metric. To tackle this issue, we propose a novel architecture with cross modality augmentation. Specifically, we first augment cross-modal data by simulating the calculation procedure of WER, i.e., substitution, deletion and insertion on both text label and its corresponding video. With these real and generated pseudo video-text pairs, we propose multiple loss terms to minimize the cross modality distance between the video and ground truth label, and make the network distinguish the difference between real and pseudo modalities. The proposed framework can be easily extended to other existing CTC based continuous SLR architectures. Extensive experiments on two continuous SLR benchmarks, i.e., RWTH-PHOENIX-Weather and CSL, validate the effectiveness of our proposed method.</p>
<div class="alert alert-note">
<div>
Please refer to this <a href="https://ustc-slr.github.io/" target="_blank" rel="noopener">project page</a> for more details.
</div>
</div></description>
</item>
<item>
<title>Hand-Model-Aware Sign Language Recognition</title>
<link>https://ustc-slr.github.io/publication/hu2021hand/</link>
<pubDate>Sun, 01 Nov 2020 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/hu2021hand/</guid>
<description></description>
</item>
<item>
<title>Semantic Boundary Detection With Reinforcement Learning for Continuous Sign Language Recognition</title>
<link>https://ustc-slr.github.io/publication/wei2020semantic/</link>
<pubDate>Tue, 02 Jun 2020 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/wei2020semantic/</guid>
<description></description>
</item>
<item>
<title>Boosting continuous sign language recognition via cross modality augmentation</title>
<link>https://ustc-slr.github.io/publication/pu2020boosting/</link>
<pubDate>Mon, 01 Jun 2020 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/pu2020boosting/</guid>
<description></description>
</item>
<item>
<title>NMFs-CSL Dataset</title>
<link>https://ustc-slr.github.io/datasets/2020_nmfs_csl/</link>
<pubDate>Wed, 01 Jan 2020 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/datasets/2020_nmfs_csl/</guid>
<description><center><img src="./Intro.jpg" border="0" width="85%"></center>
<h2>Introduction</h2>
<p class="textBlock">
This <b>NMFs-CSL</b> dataset explicitly emphasizes the importance of the non-manual features in sign language. It contains 1,067 Chinese sign words (610 confusing words (0~609), 457 normal words (610~1066)). It is collected with portable cameras on the mobile phones. The sign videos are recorded with 30<i>fps</i>. Each instance in the dataset contains RGB videos.
</p>
<h2>Download (<i><font color="#FF0000">*Important</font> </i>)</h2>
<p class="textBlock">
This <b>NMFs-CSL</b> dataset contains 1,067 Chinese sign words. It is released to universities and research institutes for research purpose only. To request the access right to the data resources, please follow the instructions below:
</p><ol>
<li>Download the <a href="./Release%20Agreement-NMFs-CSL.pdf" target="_blank"><font color="#FF0000">NMFs-CSL Dataset Release Agreement</font></a>;
</li><li>Read it carefully;
</li><li>Complete it appropriately. Note that the agreement should be signed by <font color="#FF0000">a full-time staff member</font> (that is, the student is not acceptable).
</li><li>Please scan the signed agreement and send it to (ustc_vslrg At 126.com) and CC to Prof. Zhou (zhwg At ustc.edu.cn) and the full-time staff member who sign the agreement.
</li></ol>
<p></p>
<h2>Notation</h2>
<p class="textBlock">
If you use this NMFs-CSL dataset in your research, please cite the following papers:
</p><ul>
<li>Hezhen Hu, Wengang Zhou, Junfu Pu, and Houqiang Li, "Global-Local Enhancement Network for NMF-Aware Sign Language Recognition," <i>ACM Trans. Multimedia Comput. Commun. Appl. (TOMM)</i>, 2021.</li>
<li>Hezhen Hu, Wengang Zhou, Junfu Pu, and Houqiang Li, "Hand-Model-Aware Sign Language Recognition," <i>AAAI Conference on Artificial Intelligence (AAAI)</i>, 2021.</li>
<li>Hezhen Hu, Weichao Zhao, Wengang Zhou, and Houqiang Li, "SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding," <i>IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI)</i>, 2023.</li>
<li>Hezhen Hu, Weichao Zhao, Wengang Zhou, Yuechen Wang, Houqiang Li, "SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition," <i>International Conference on Computer Vision (ICCV)</i>, 2021.</li>
</ul>
<p></p>
<h2>Contact</h2>
<p class="textBlock">
If you have any questions about the dataset and our papers, please feel free to contact us:
</p><ul>
<li><a href="http://staff.ustc.edu.cn/~lihq/English.html" target="_blank">Houqiang Li</a>, Professor, USTC, lihq AT ustc.edu.cn</li>
<li><a href="http://staff.ustc.edu.cn/~zhwg" target="_blank">Wengang Zhou</a>, Professor, USTC, zhwg AT ustc.edu.cn</li>
<li><a href="https://ustc-slr.github.io/" target="_blank">Weichao Zhao</a>, Ph.D Student, USTC, saruka AT mail.ustc.edu.cn</li>
</ul>
<p></p></description>
</item>
<item>
<title>Continuous sign language recognition via reinforcement learning</title>
<link>https://ustc-slr.github.io/publication/zhang2019continuous/</link>
<pubDate>Fri, 01 Nov 2019 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/zhang2019continuous/</guid>
<description></description>
</item>
<item>
<title>Deep grammatical multi-classifier for continuous sign language recognition</title>
<link>https://ustc-slr.github.io/publication/wei2019deep/</link>
<pubDate>Fri, 01 Nov 2019 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/wei2019deep/</guid>
<description></description>
</item>
<item>
<title>Dynamic pseudo label decoding for continuous sign language recognition</title>
<link>https://ustc-slr.github.io/publication/zhou2019dynamic/</link>
<pubDate>Fri, 01 Nov 2019 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/zhou2019dynamic/</guid>
<description></description>
</item>
<item>
<title>Spatial-temporal multi-cue network for continuous sign language recognition</title>
<link>https://ustc-slr.github.io/publication/zhou2020spatial/</link>
<pubDate>Fri, 01 Nov 2019 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/zhou2020spatial/</guid>
<description></description>
</item>
<item>
<title>Iterative alignment network for continuous sign language recognition</title>
<link>https://ustc-slr.github.io/publication/pu2019iterative/</link>
<pubDate>Fri, 01 Feb 2019 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/pu2019iterative/</guid>
<description></description>
</item>
<item>
<title>Hierarchical LSTM for sign language translation</title>
<link>https://ustc-slr.github.io/publication/guo2018hierarchical/</link>
<pubDate>Thu, 01 Nov 2018 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/guo2018hierarchical/</guid>
<description></description>
</item>
<item>
<title>Video-based sign language recognition without temporal segmentation</title>
<link>https://ustc-slr.github.io/publication/huang2018video/</link>
<pubDate>Thu, 01 Nov 2018 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/huang2018video/</guid>
<description></description>
</item>
<item>
<title>Connectionist Temporal Fusion for Sign Language Translation</title>
<link>https://ustc-slr.github.io/publication/wang2018connectionist/</link>
<pubDate>Mon, 15 Oct 2018 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/wang2018connectionist/</guid>
<description></description>
</item>
<item>
<title>Attention-Based 3D-CNNs for Large-Vocabulary Sign Language Recognition</title>
<link>https://ustc-slr.github.io/publication/huang2018attention/</link>
<pubDate>Sun, 16 Sep 2018 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/huang2018attention/</guid>
<description></description>
</item>
<item>
<title>Dilated convolutional network with iterative optimization for continuous sign language recognition</title>
<link>https://ustc-slr.github.io/publication/pu2018dilated/</link>
<pubDate>Thu, 01 Feb 2018 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/pu2018dilated/</guid>
<description></description>
</item>
<item>
<title>Sign Language Recognition with Multi-modal Features</title>
<link>https://ustc-slr.github.io/publication/pu2016sign2/</link>
<pubDate>Sun, 27 Nov 2016 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/pu2016sign2/</guid>
<description></description>
</item>
<item>
<title>Sign language recognition based on adaptive HMMs with data augmentation</title>
<link>https://ustc-slr.github.io/publication/guo2016sign/</link>
<pubDate>Wed, 28 Sep 2016 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/guo2016sign/</guid>
<description></description>
</item>
<item>
<title>Sign language recognition with long short-term memory</title>
<link>https://ustc-slr.github.io/publication/liu2016sign/</link>
<pubDate>Wed, 28 Sep 2016 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/liu2016sign/</guid>
<description></description>
</item>
<item>
<title>Chinese sign language recognition with adaptive HMM</title>
<link>https://ustc-slr.github.io/publication/zhang2016chinese/</link>
<pubDate>Mon, 11 Jul 2016 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/zhang2016chinese/</guid>
<description></description>
</item>
<item>
<title>Sign Language Recognition Based on Trajectory Modeling with HMMs</title>
<link>https://ustc-slr.github.io/publication/pu2016sign/</link>
<pubDate>Sun, 03 Jan 2016 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/pu2016sign/</guid>
<description></description>
</item>
<item>
<title>A new system for Chinese sign language recognition</title>
<link>https://ustc-slr.github.io/publication/zhang2015new/</link>
<pubDate>Sun, 12 Jul 2015 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/zhang2015new/</guid>
<description></description>
</item>
<item>
<title>Sign Language Recognition using Real-Sense</title>
<link>https://ustc-slr.github.io/publication/huang2015sign2/</link>
<pubDate>Sun, 12 Jul 2015 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/huang2015sign2/</guid>
<description></description>
</item>
<item>
<title>Sign Language Recognition using 3D convolutional neural networks</title>
<link>https://ustc-slr.github.io/publication/huang2015sign/</link>
<pubDate>Mon, 29 Jun 2015 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/huang2015sign/</guid>
<description></description>
</item>
<item>
<title>CSL & SLR500 Dataset</title>
<link>https://ustc-slr.github.io/datasets/2015_csl/</link>
<pubDate>Thu, 01 Jan 2015 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/datasets/2015_csl/</guid>
<description><h2>Download (<i><font color="#FF0000">*Important</font> </i>)</h2>
<p class="textBlock">
The CSL databases are released to universities and research institutes for research purpose only. To request the access right to the data resources, please follow the instructions below:
</p><ol>
<li>Download the <a href="./Release-Agreement-csl2015.pdf" target="_blank"><font color="#FF0000">CSL Dataset Release Agreement</font></a>;
</li><li>Read it carefully;
</li><li>Complete it appropriately. Note that the agreement should be signed by <font color="#FF0000">a full-time staff member</font> (that is, the student is not acceptable).
</li><li>Please scan the signed agreement and send it to (ustc_vslrg At 126.com) and CC to Prof. Zhou (zhwg At ustc.edu.cn) and the full-time staff member who sign the agreement.
</li></ol>
<p></p>
<h2>Introduction</h2>
<p class="textBlock">
We have two Chinese sign language datasets for isolated Sign Language Recognition and continuous Sign Language Recognition, respectively. Both datasets are collected with Kinect 2.0 by 50 signers. Each signer perfors 5 times for every word (sentence). The sign videos are recorded with 30<i>fps</i>. The distance between the signers and Kinect is about 1.5 meters. Each instance in both datasets contains <b>RGB videos</b>, <b>depth videos</b>, and <b>3D joints information</b> of the signer.
<br><br>
Since the dataset is recorded with Microsoft Kinect, there are three data modalities available:
</p><ol>
<li>RGB videos with resolution of 1280 x 720 pixels and frame rate of 30 fps.</li>
<li>Depth videos with resolution of 512x 424 pixels and frame rate of 30 fps.</li>
<li>Twenty-five skeleton joints locations of each frame.</li>
</ol>
<p></p>
<h2>Isolated SLR</h2>
<p class="textBlock">
The isolated SLR dataset contains 500 Chinese sign words. Each sign video is performed by 50 signers with 5 times. Hence, there are 250 instances for each sign word.
</p>
<p class="textBlock">
If you use this Chinese isolated SLR dataset in your research, please consider citing the following papers:
</p><ul>
<li>Hezhen Hu, Weichao Zhao, Wengang Zhou, and Houqiang Li, "SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding," <i>IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI)</i>, 2023.</li>
<li>Weichao Zhao, Hezhen Hu, Wengang Zhou, Jiaxin Shi, and Houqiang Li, "BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenization," <i>Conference on Artificial Intelligence (AAAI)</i>, 2023.</li>
<li>Jie Huang, Wengang Zhou, Houqiang Li, and Weiping Li, "Attention based 3D-CNNs for Large-Vocabulary Sign Language Recognition, <i>IEEE Trans. Circuits Syst. Video Technol. (TCSVT)</i>, 2018.</li>
</ul>
<p></p>
<h2>Continuous SLR</h2>
<p class="textBlock">
The corpus of continuous SLR dataset contains 100 Chinese sentence. There are 250 instances (50signers x 5times) for each sentence.
</p>
<p class="textBlock">
If you use this Chinese continuous SLR dataset in your research, please consider citing the following papers:
</p><ul>
<li>Junfu Pu, Wengang Zhou, and Houqiang Li, "Iterative Alignment Network for Continuous Sign Language Recognition," <i>Conference on Computer Vision and Pattern Recognition (CVPR)</i>, 2019.</li>
<li>Hao Zhou, Wengang Zhou, and Houqiang Li, "Dynamic Pseudo Label Decoding for Continuous Sign Language Recognition," <i>International Conference on Multimedia and Expo (ICME)</i>, 2019.</li>
<li>Jie Huang, Wengang Zhou, Qilin Zhang, Houqiang Li and Weiping Li, "Video-based Sign Language Recognition without Temporal Segmentation," <i>AAAI Conference on Artificial Intelligence (AAAI)</i>, 2018.</li>
</ul>
<p></p>
<p class="textBlock">
Besides, you can refer to the following papers for continuous SLR published by our group:
</p><ul>
<li>Hezhen Hu, Weichao Zhao, Wengang Zhou, and Houqiang Li, "SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding," <i>IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI)</i>, 2023.</li>
<li>Hezhen Hu, Junfu Pu, Wengang Zhou, and Houqiang Li, "Collaborative Multilingual Continuous Sign Language Recognition: A Unified Framework," <i>IEEE Trans. Multimedia (TMM)</i>, 2022.</li>
<li>Junfu Pu, Wengang Zhou, Hezhen Hu, and Houqiang Li, "Boosting Continuous Sign Language Recognition via Cross Modality Augmentation," <i>ACM International Conference on Multimedia (ACM MM)</i>, 2020.</li>
</ul>
<p></p>
<h2>Contact</h2>
<p class="textBlock">
If you have any questions about the dataset and our papers, please feel free to contact us:
</p><ul>
<li><a href="http://staff.ustc.edu.cn/~lihq/English.html" target="_blank">Houqiang Li</a>, Professor, USTC, lihq AT ustc.edu.cn</li>
<li><a href="http://staff.ustc.edu.cn/~zhwg" target="_blank">Wengang Zhou</a>, Professor, USTC, zhwg AT ustc.edu.cn</li>
<li><a href="https://ustc-slr.github.io/" target="_blank">Weichao Zhao</a>, Ph.D Student, USTC, saruka AT mail.ustc.edu.cn</li>
</ul>
<p></p></description>
</item>
<item>
<title>A Threshold-based HMM-DTW Approach for Continuous Sign Language Recognition</title>
<link>https://ustc-slr.github.io/publication/zhang2014threshold/</link>
<pubDate>Thu, 10 Jul 2014 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/publication/zhang2014threshold/</guid>
<description></description>
</item>
<item>
<title></title>
<link>https://ustc-slr.github.io/admin/config.yml</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/admin/config.yml</guid>
<description></description>
</item>
<item>
<title></title>
<link>https://ustc-slr.github.io/people/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://ustc-slr.github.io/people/</guid>
<description></description>
</item>
</channel>
</rss>