WMT Shared Task 22

[cleong110]

https://aclanthology.org/2022.wmt-1.71/

• sync, pull and merge master first!
• Search for the correct citation on Semantic Scholar
• Make a new branch ("You should always branch out from master")
• Add citation to references.bib. If dataset, prepend with dataset:. Exclude wordy abstracts. (better BibTex extension to Zotero can exclude keys)
• Check for egregious {} in the bibtex
• write a summary and add to the appropriate section in index.md.
• Make sure the citation keys match.
• Add a newline after each sentence in a paragraph. Still shows up as one paragraph but makes git stuff easier.
• ChatGPT 3.5 can suggest rewrites and improve writing.
• Check if acronyms are explained
• Copy-Paste into https://dillinger.io/, see if it looks OK
• Make a PR from the branch on my fork to master on the source repo

PR:

• sync master of both forks
• git pull master on local
• git merge master on branch
• git push
• THEN make the PR

Writing/style:

• try to describe what they did, not what the general process is.
• Don't have to describe what's in a repo
• something like "Three Letter Acronym (TLA)" is how you introduce acronyms
• Look through the Style Guide on the README
• Evaluations on X and Y datasets should have a the. Evaluation on the X and Y datasets"

[cleong110]

Citation is already in references.bib, muller-etal-2022-findings

[cleong110]

https://github.com/cleong110/sign-language-processing.github.io/tree/paper/muller-etal-2022-findings branch

[cleong110]

Salient facts to know from the Findings:
7 submissions from DSGS-> German, 0 the other way.

Overall, all systems perform poorly in our shared task, as there is an extreme difference in average score between all systems and the human reference translation

The best submitted system in the best case achieves an average score of about 4 out of 100, which indicates that current automatic translations are not usable in practice

The best system in the automatic evaluation depends on the evaluation metric.

BLEU, chrF, BLEURT

participants probably used mainly BLEU as the metric for development, except DFKI-SLT who reported that they used chrF because BLEU was always zero.

The correlation between human rankings and automatic metrics is delicate because we only have seven data points. The metric that correlates best with human scores at system level is BLEU (r = 0.510, ρ = 0.571) followed by chrF (r = 0.508, ρ = 0.214). BLEURT shows only a weak correlation with r = 0.314 and ρ = 0.286. In our scenario, translation quality is really low, and the sentences that have been properly translated are very short (e.g. Bis nächste Woche.). In this case, n-gram matching met

[cleong110]

Submission: "Experimental Machine Translation of the Swiss German Sign Language via 3D augmentation of body keypoints."

@inproceedings{hufe-avramidis-2022-experimental,
    title = "Experimental Machine Translation of the {S}wiss {G}erman Sign Language via 3{D} Augmentation of Body Keypoints",
    author = "Hufe, Lorenz  and
      Avramidis, Eleftherios",
    editor = {Koehn, Philipp  and
      Barrault, Lo{\"\i}c  and
      Bojar, Ond{\v{r}}ej  and
      Bougares, Fethi  and
      Chatterjee, Rajen  and
      Costa-juss{\`a}, Marta R.  and
      Federmann, Christian  and
      Fishel, Mark  and
      Fraser, Alexander  and
      Freitag, Markus  and
      Graham, Yvette  and
      Grundkiewicz, Roman  and
      Guzman, Paco  and
      Haddow, Barry  and
      Huck, Matthias  and
      Jimeno Yepes, Antonio  and
      Kocmi, Tom  and
      Martins, Andr{\'e}  and
      Morishita, Makoto  and
      Monz, Christof  and
      Nagata, Masaaki  and
      Nakazawa, Toshiaki  and
      Negri, Matteo  and
      N{\'e}v{\'e}ol, Aur{\'e}lie  and
      Neves, Mariana  and
      Popel, Martin  and
      Turchi, Marco  and
      Zampieri, Marcos},
    booktitle = "Proceedings of the Seventh Conference on Machine Translation (WMT)",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates (Hybrid)",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.wmt-1.95",
    pages = "983--988",
    abstract = "This paper describes the participation of DFKI-SLT at the Sign Language Translation Task of the Seventh Conference of Machine Translation (WMT22). The system focuses on the translation direction from the Swiss German Sign Language (DSGS) to written German. The original videos of the sign language were analyzed with computer vision models to provide 3D body keypoints. A deep-learning sequence-to-sequence model is trained on a parallel corpus of these body keypoints aligned to written German sentences. Geometric data augmentation occurs during the training process. The body keypoints are augmented by artificial rotation in the three dimensional space. The 3D-transformation is calculated with different angles on every batch of the training process.",
}

The authors describe their system as having three main modules. In the first, SL images are converted into intermediate pose keypoint representations; the second module employs data augmentation (geometrical transformations) to increase sample efficiency and decrease the effect of spurious feature correlations; and the third employs a Transformer network to perform translation.

So video->pose->text

[cleong110]

Submission: Tackling Low-Resource Sign Language Translation: UPC at WMT-SLT 22.

https://aclanthology.org/2022.wmt-1.97/

• Transformer based sequence-to-sequence model,
• Pretrain on RWTH-PHOENIX-WEATHER 2014, no "T" apparently, but did not do anything.
• The best results were obtained by only training on the FocusNews dataset.
• MediaPipe Holistic was used, re-extracting the features using the pose library by Moryossef and Müller (2021a).
• augmentation using pose libary augmentation functions such as rotation, scaling and shear)
• For the text side Sentencepiece was used to generate vocabularies
• Fairseq

Amit Moryossef and Mathias Müller. 2021a. poseformat: Library for viewing, augmenting, and handling .pose files. https://github.com/ AmitMY/pose-format.

So video->pose->text (via transformer)

[cleong110]

submission: DFKI-MLT at WMT-SLT22: Spatio-temporal Sign Language Representation and Translation. In Proceedings of the Seventh Conference on Machine Translation

Yasser Hamidullah, Josef van Genabith, and Cristina España-Bonet. 2022. DFKI-MLT at WMT-SLT22: Spatio-temporal Sign Language Representation and Translation. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

https://aclanthology.org/2022.wmt-1.94/

The main idea behind the DFKI-MLT approach is to learn feature representation and translation in a single model, and to train them together.

The visual component is based on a ResNet (Hara et al., 2017).
In particular, the visual encoding in the submitted system consists of the original 3D ResNet10 with output conversion. The conversion creates a sequence of vectors from the single output vector to adapt to the Transformer encoder input. The visual vector is projected through a linear layer which is connected directly to the language block.

The language block is a simple Transformer.

The training is end-to-end, aiming to force the visual block to take into account the language representation when building the visual embedding.

So this one is video->text, no intermediate. 3D CNNs projected to text transformer

[cleong110]

Submission by MSMUNICH (Dey et al., 2022)

Subhadeep Dey, Abhilash Pal, Cyrine Chaabani, and Oscar Koller. 2022. Clean Text and FullBody Transformer: Microsoft’s Submission to the WMT22 Shared Task on Sign Language Translation. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

sequence-to-sequence Transformer model

based on an existing model pretrained on the BSL1K dataset (Varol et al., 2021)

pretrained I3D model.

Fairseq

Looks like it's video->text directly. I3D to Fairseq somehow

[cleong110]

6.2 Submission by LATTIC (Shi et al., 2022)

The system submitted by LATTIC is a Transformer-based sequence-to-sequence model which uses as input visual representations derived from an Inflated 3D ConvNet (I3D) (Carreira and Zisserman, 2017) and text as the target

pretrained on WLASL dataset:li2020word

The I3D model is first trained on Kinetics, an action recognition dataset (Carreira and Zisserman, 2017), then it is trained for isolated sign language recognition.

The system is developed from scratch, without the use of existing MT software, and has a Transformer architecture (Vaswani et al., 2017).

For the spoken language side Sentencepiece (Kudo and Richardson, 2018) was used to generate a vocabulary of 18k tokens.

So video->Text, I3D for inputs, pretrained on Kinetics (and then WLASL?)

[cleong110]

6.1 Submission by UZH (baseline system)

The underlying sequence-to-sequence toolkit is Sockeye (Hieber et al., 2022) which is based on Pytorch (Paszke et al., 2019). We adapted Sockeye so that it supports encoding or decoding continuous vectors instead of discrete sequences of tokens.

our system is a pose-to-text translation model that reads a sequence of pose frames and converts them to the model size with a simple learned projection. The baseline does not involve pretraining or additional data and is therefore a constrained submission.

[cleong110]

Hmmm, can we use this figure?