Repositories list

• warc2text-runner

  Public
  Scripts for parallelized extraction of plain texts from WARC archieves. Aiming at common and reproducible extraction approach.

  HTML

  •0•3•5•1•Updated Oct 10, 2024Oct 10, 2024

• bitextor-mt-models

  Public
  Shell

  •0•1•3•0•Updated Oct 8, 2024Oct 8, 2024

• OpusPocus

  Public
  Marian machine translation training pipeline for thousands of models

  Python

  •0•2•22•1•Updated Oct 8, 2024Oct 8, 2024

• data-analytics-tool

  Public
  Data Analytics Tool

  JavaScript

  •1•8•0•0•Updated Oct 7, 2024Oct 7, 2024

• cc-download

  Public
  Shell

  •0•0•0•0•Updated Oct 6, 2024Oct 6, 2024

• bitextor-slurm

  Public
  Scripts for running bitextor jobs

  Shell

  •1•0•0•0•Updated Sep 26, 2024Sep 26, 2024

• OpusTrainer

  Public
  Curriculum training

  machine-translationtraining-pipeline

  Python

  •

  MIT License

  •5•15•19•0•Updated Sep 14, 2024Sep 14, 2024

• OpusCleaner

  Public
  OpusCleaner is a web interface that helps you select, clean and schedule your data for training machine translation models.

  machine-translationdata-cleaning

  Python

  •13•46•56•1•Updated Sep 7, 2024Sep 7, 2024

• monotextor-slurm

  Public
  Set of scripts to run monotextor-like pipeline under slurm HPCs

  Rust

  •

  GNU General Public License v3.0

  •0•2•0•0•Updated Sep 5, 2024Sep 5, 2024

• ia-download

  Public
  Internet archive downloader

  Jupyter Notebook

  •0•2•1•0•Updated Aug 7, 2024Aug 7, 2024

• HPLT-WP4

  Public
  Information and pipelines on WP4: language models training

  Python

  •

  Creative Commons Zero v1.0 Universal

  •2•1•0•0•Updated Jul 11, 2024Jul 11, 2024

• sacremoses

  Public
  Python port of Moses tokenizer, truecaser and normalizer

  nlptokenizermachine-translation

  Python

  •

  MIT License

  •59•486•26•5•Updated May 26, 2024May 26, 2024

• document-aligner

  Public
  tf/idf-based document aligner from Bitextor

  C++

  •

  Apache License 2.0

  •0•0•0•1•Updated Mar 19, 2024Mar 19, 2024

• OPUS-MT-dashboard

  Public
  PHP

  •

  MIT License

  •1•0•0•0•Updated Mar 9, 2024Mar 9, 2024

• HPLT-MT-Models

  Public
  This contains the configuration and scripts for HPLT MT model releases.

  Python

  •0•4•1•0•Updated Mar 6, 2024Mar 6, 2024

• monolingual-multilingual-instruction-tuning

  Public
  Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca

  large-language-modelsinstruction-tuninginstruction-data

  Python

  •0•8•0•0•Updated Mar 6, 2024Mar 6, 2024

• OpusFilter

  Public
  OpusFilter - Parallel corpus processing toolkit

  Python

  •

  MIT License

  •18•0•0•0•Updated Jan 3, 2024Jan 3, 2024

• clianer

  Public
  A lightweight command-line frontend to OpusCleaner

  Python

  •

  MIT License

  •1•0•0•0•Updated Nov 27, 2023Nov 27, 2023

• paracrawl-dashboard

  Public
  Make-shift interface for managing Paracrawl processing and exploring its outputs

  HTML

  •1•0•0•0•Updated Oct 10, 2023Oct 10, 2023

• MT-winterschool-2023

  Public
  0•1•0•0•Updated Feb 7, 2023Feb 7, 2023