Repositories list

• HPLT-textpipes

  Public
  Step-by-step schematic description of data processing in HPLT

  Python

  •1•0•4•0•Updated Aug 4, 2025Aug 4, 2025

• HPLT-WP4

  Public
  Information and pipelines on WP4: language models training

  Jupyter Notebook

  •

  Creative Commons Zero v1.0 Universal

  •3•3•5•0•Updated Aug 4, 2025Aug 4, 2025

• data-analytics-tool

  Public
  HPLT Analytics

  JavaScript

  •

  GNU General Public License v3.0

  •3•14•1•1•Updated Aug 4, 2025Aug 4, 2025

• HPLT-MT-Models

  Public
  This contains the configuration and scripts for HPLT MT model releases.

  Python

  •0•8•2•0•Updated Aug 4, 2025Aug 4, 2025

• warc2text-runner

  Public
  Scripts for parallelized extraction of plain texts from WARC archieves. Aiming at common and reproducible extraction approach.

  Jupyter Notebook

  •1•4•4•0•Updated Jul 24, 2025Jul 24, 2025

• monotextor-slurm

  Public
  Set of scripts to run monotextor-like pipeline under slurm HPCs

  Rust

  •

  GNU General Public License v3.0

  •0•3•0•0•Updated Jul 15, 2025Jul 15, 2025

• OpusCleaner

  Public
  OpusCleaner is a web interface that helps you select, clean and schedule your data for training machine translation models.

  machine-translationdata-cleaning

  Python

  •15•51•56•1•Updated Jul 9, 2025Jul 9, 2025

• OpusTrainer

  Public
  Curriculum training

  machine-translationtraining-pipeline

  Python

  •

  MIT License

  •6•18•19•0•Updated Jun 25, 2025Jun 25, 2025

• release2_inspection

  Public
  Jupyter Notebook

  •7•1•1•0•Updated May 27, 2025May 27, 2025

• OpusPocus

  Public
  Marian machine translation training pipeline for thousands of models

  Python

  •0•2•20•0•Updated May 16, 2025May 16, 2025

• hplt-e

  Public
  Jupyter Notebook

  •0•1•4•0•Updated May 13, 2025May 13, 2025

• bitextor-mt-models

  Public
  Shell

  •0•1•3•0•Updated Feb 24, 2025Feb 24, 2025

• ia-download

  Public
  Internet archive downloader

  Jupyter Notebook

  •0•2•2•0•Updated Jan 26, 2025Jan 26, 2025

• cc-download

  Public
  Shell

  •0•0•0•0•Updated Jan 26, 2025Jan 26, 2025

• bitextor-slurm

  Public
  Scripts for running bitextor jobs

  Shell

  •1•0•1•0•Updated Jan 20, 2025Jan 20, 2025

• monolingual-multilingual-instruction-tuning

  Public
  Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca

  large-language-modelsinstruction-tuninginstruction-data

  Python

  •1•10•0•0•Updated Nov 2, 2024Nov 2, 2024

• sacremoses

  Public
  Python port of Moses tokenizer, truecaser and normalizer

  nlptokenizermachine-translation

  Python

  •

  MIT License

  •60•495•27•4•Updated May 26, 2024May 26, 2024

• document-aligner

  Public
  tf/idf-based document aligner from Bitextor

  C++

  •

  Apache License 2.0

  •0•0•0•1•Updated Mar 19, 2024Mar 19, 2024

• OPUS-MT-dashboard

  Public
  PHP

  •

  MIT License

  •1•0•0•0•Updated Mar 9, 2024Mar 9, 2024

• OpusFilter

  Public
  OpusFilter - Parallel corpus processing toolkit

  Python

  •

  MIT License

  •24•0•0•0•Updated Jan 3, 2024Jan 3, 2024

• clianer

  Public
  A lightweight command-line frontend to OpusCleaner

  Python

  •

  MIT License

  •1•0•0•0•Updated Nov 27, 2023Nov 27, 2023

• paracrawl-dashboard

  Public
  Make-shift interface for managing Paracrawl processing and exploring its outputs

  HTML

  •1•0•0•0•Updated Oct 10, 2023Oct 10, 2023

• MT-winterschool-2023

  Public
  0•1•0•0•Updated Feb 7, 2023Feb 7, 2023