We developed a Python package called LinguAligner, a comprehensive corpus translation and alignment pipeline designed to facilitate the translation of corpora across different languages. It translates corpora using machine translation and aligns the translated annotations with their corresponding translated text. Initially developed for the automatic translation of ACE-2005 into Portuguese , LinguAligner has since been adapted into a versatile package for effortless translation of other corpora.
It is composed of two main components:
- Text translation: We support DeepL Translator, Google Translator and Microsoft Translators APIs.
- Annotations alignments: We developed an annotation alignment pipeline that uses several alignment techniques to align the translated annotations within the translated text.
You can access the LinguAligner Python package here.
The ACE-2005-PT corpus (Portuguese translation produced with LinguAligner), was published by the Linguistic Data Consortium. For more details, visit the LDC catalog.
Our pipeline is composed of a total of five annotation alignment components:
- Lemmatization
- Multiple word translation
- BERT-based word aligner
- Gestalt Patter Matching
- Levenstein distance
The pipeline operates sequentially, meaning that annotations aligned by earlier methods are not addressed by subsequent pipeline elements. According to our experiments, the list above corresponds to the best order sequence.
- Translate Corpora You can use the Translation APIs or can translate your corpus with an external tool An API key is needed to use some of the Translation APIs.
from LinguAligner import translation
# Google Translator
translator = translation.GoogleTranslator(source_lang="en", target_lang="pt")
translated_text = translator.translate("The soldiers were ordered to fire their weapons")
# DeepL Translator
translator = translation.DeepLTranslator(source_lang="en", target_lang="pt", key="DEEPL_KEY")
translated_text = translator.translate("The soldiers were ordered to fire their weapons")
# Microsoft Translator
translator = translation.MicrosoftTranslator(source_lang="en", target_lang="pt", key="MICROSOFT_TRANSLATOR_KEY")
translated_text = translator.translate("The soldiers were ordered to fire their weapons")
print(translated_text)- Run the Annotation Alignment Pipeline Users can select the aligners they intend to use and must indicate the path for the alignment resources for each alignment component, such as multiple translations of annotations, previously calculated lemmas, synonyms, etc.
from LinguAligner import AlignmentPipeline
"""
(By default, the first method used is string matching. If unsuccessful, the alignment pipeline is employed.)
Methods:
- lemma: Lemmatization
- M_Trans: Multiple Translations of a word
- word_aligner: mBERT-based word aligner
- gestalt: Gestalt pattern matching (character-based)
- levenshtein: Levenshtein distance (character-based)
"""
config= {
"pipeline": [ "lemma", "M_Trans", "word_aligner","gestalt","leveinstein"], # can be changed according to the desired pipeline
"spacy_model": "pt_core_news_lg", # change according to the language
"WAligner_model": "bert-base-multilingual-uncased", # needed for word_aligner
}
aligner = AlignmentPipeline(config)
src_sentence = "The soldiers were ordered to fire their weapons."
src_annotation = "fire"
translated_sentence = "Os soldados receberam ordens para disparar as suas armas."
translated_annotation = "incêndio"
target_annotation = aligner.align_annotation(src_sentence, src_annotation, translated_sentence, translated_annotation)
print(target_annotation)
>>> "disparar"For example, in the sentence 'The soldiers were ordered to fire their weapons,' the word 'fire' was annotated in the source corpus. However, when this sentence is translated to 'Os soldados receberam ordens para disparar as suas armas,' the word 'fire' is translated to 'incêndio' (fire as a noun) in isolation, and to 'disparar' (as a verb) in the translated sentence.
Note
To use the M_trans method, multiple translations of the annotations must be computed beforehand and passed as an argument to the align_annotation function. These translations should contained in a Python dictionary, where the source annotation serves as the key, and the corresponding value is a list of alternative translations. You can generate this dictionary using the following code (need a MICROSOFT_TRANSLATOR_KEY):
from LinguAligner import translation
translator = translation.MicrosoftTranslator(source_lang="en", target_lang="pt", auth_key="MICROSOFT_TRANSLATOR_KEY")
lookupTable = {}
annotations_list = ["war","land","fire"]
for word in annotations_list:
lookupTable[word] = translator.getMultipleTranslations(word) # change the language codes according to the desired languages
# Then, pass the lookupTable to the align_annotation method
x = aligner.align_annotation("The soldiers were ordered to fire their weapons","fire", "Os soldados receberam ordens para disparar as suas armas","incêndio",lookupTable)
To measure the effectiveness of the alignment pipeline we tested it on ACE-2005 corpus. Manual alignments were conducted on the entire ACE-2005-PT test set, which includes 1,310 annotations. These alignments were performed by a linguist expert to ensure high-quality annotations, following the same annotation guidelines of the original ACE-2005 corpus. Then we compare the manual alignments against the ones generated by our pipeline.
The evaluation results are presented in Table 1:
Table 1: Evaluation Results by pipeline component
This project is licensed under the MIT License.
Comming Soon.