v2.0.0

• New model architecture (UnifiedMetric) inspired by UniTE.
  - This model uses cross-encoding (similar to BLEURT), works with and without references and can be trained in a multitask setting. This model is also implemented in a very flexible way where we can decide to train using just source and MT, reference and MT or source, MT and reference.

• New encoder models RemBERT and XLM-RoBERTa-XL

• New training features:
  - System-level accuracy (Kocmi et al, 2021) reported during validation (only if validation files has a system column).
  - Support for multiple training files (each file will be loaded at the end of the corresponding epoch): This is helpful to train with large datasets and to train following a curriculum.
  - Support for multiple validation files: Before we were using 1 single validation file with all language pairs concatenated which has an impact in correlations. With this change we now can have 1 validation file for each language and correlations will be averaged over all validation sets. This also allows for the use of validation files where the ground truth scores are in different scales.
  - Support to HuggingFace Hub: Models can now be easily added to HuggingFace Hub and used directly using the CLI

• With this release we also add New models from WMT 22:
  1) We won the WMT 22 QE shared task: Using UnifiedMetric it should be easy to replicate our final system, nonetheless we are planning to release the system that was used: wmt22-cometkiwi-da which performs strongly both on data from the QE task (MLQE-PE corpus) and on data from metrics task (MQM annotations).
  2) We were 2nd in the Metrics task (1st place was MetricXL a 6B parameter metric trained on top of mT5-XXL). Our new model wmt22-comet-da was part of the ensemble used to secure our result.

If you are interested in our work from this year please read the following paper:

• CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task
• COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task

And the corresponding findings papers:

• Findings of the WMT 2022 Shared Task on Quality Estimation
• Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust

Special thanks to all the involved people: @mtreviso @nunonmg @glushkovato @chryssa-zrv @jsouza @DuarteMRAlves @Catarinafarinha @cmaroti

v1.1.3

Same as v1.1.2 but we bumped some requirements in order to be easier to use COMET on Windows and Apple M1.

Version 1.1.2

Just minor requirement updates to avoid installation errors described in #82

Version 1.1.1

1. comet-compare to support multiple system comparisons.
2. Bugfix: Broken link for wmt21-comet-qe-da (#78)
3. Bugfix: protobuf dependency (#82)
4. New models from Cometinho EAMT 22 paper (eamt22-cometinho-da & eamt22-comet-prune-da)

Breaking Changed

comet-compare does not support -xand -y flags. Now it receives a single flag -t with multiple arguments for multiples systems.

Before:

comet-compare -s src.de -x hyp1.en -y hyp2.en -r ref.en

After:

comet-compare -s src.de -t hyp1.en hyp2.en -r ref.en

Contributors

• @erip (#69, #70)
• @SamuelLarkin (#74)
• @Joao-Maria-Janeiro (#75, #77)

Full Changelog: v1.0.1...v1.1.0

Version 1.1.0

1. Updated documentation
2. Updated Pytorch Lightning version to avoid security vulnerabilities (Untrusted Data & Code Injection)
3. Inspired by Amrhein et al, 2022 we added the comet-mbr command for fast Minimum Bayes Risk Decoding.
4. New encoder models

What's Changed

• Fix minor typo in exception message by @alvations in #57
• Adds --quiet flag by @Remorax in #58
• Bug fix of num_workers. by @devrimcavusoglu in #63
• fix encoding issues for Windows users by @erip in #68

New Contributors

• @alvations made their first contribution in #57
• @Remorax made their first contribution in #58
• @devrimcavusoglu made their first contribution in #63
• @erip made their first contribution in #68

Full Changelog: v1.0.1...v1.1.0

Version 1.0.1

Scipy missing from dependencies list.

Version 1.0.0

What's new?

1. comet-compare command for statistical comparison between two models
2. comet-score with multiple hypothesis/systems
3. Embeddings caching for faster inference (thanks to @jsouza).
4. Length Batching for faster inference (thanks to @CoderPat)
5. Integration with SacreBLEU for dataset downloading (thanks to @mjpost)
6. Monte-carlo Dropout for uncertainty estimation (thanks to @glushkovato and @chryssa-zrv)
7. Some code refactoring

Hopefully, this version is also easier to install than the previous one that relied on fairseq.

Version 0.1.0

• We now use Poetry to solve dependency issues.
• Removed LASER encoder and FastBPE dependencies (Windows users can now run COMET)
• Removed references requirements for QE models