Auto detect language feature ?

[Utsaww]

It's a really good project working out of the box, much appreciated man!
I was wondering if language auto detection feature is there it will be really helpful.

[thammegowda]

I agree language ID would be a great feature.
Many available language ID models (e.g. https://fasttext.cc/docs/en/language-identification.html) recognize fewer than 200 languages (they fall short of NLLB models). Are you familiar with any good lang ID detection models that recognize all 200 languages in NLLB?

Pull requests will be greatly appreciated!!

[Utsaww]

I am working on fasttext Language identification as it has 176 language identification, for now I guess this works, if you are good to go, then I will surely work and create a pull request.

[thammegowda]

@Utsaww yes, please! Sorry I missed replying to this message.

Suggestion on how to integrate:
here ...

nllb-serve/nllb_serve/app.py

Line 128 in 024f703

src_lang = args.get('src_lang') or def_src_lang

if src_lang == '[auto]':
  src_lang = <lang_id_detection>(sources)

[omercandemir]

This is how I detect lang with fasttext. But I haven't tried it yet. I think I need to match the Langids.

    def language_detection_fasttext(self, text: str) -> str:
        """
        Given a text, detects the language code and returns the ISO language code. It supports 176 languages. Uses
        the fasttext model for language detection:
        https://fasttext.cc/blog/2017/10/02/blog-post.html
        https://fasttext.cc/docs/en/language-identification.html


        """
        if self._fasttext_lang_id is None:
            import fasttext
            fasttext.FastText.eprint = lambda x: None   #Silence useless warning: https://github.com/facebookresearch/fastText/issues/1067
            model_path = os.path.join(self._cache_folder, 'lid.176.ftz')
            if not os.path.exists(model_path):
                http_get('https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.ftz', model_path)
            self._fasttext_lang_id = fasttext.load_model(model_path)

        return self._fasttext_lang_id.predict(text.lower().replace("\r\n", " ").replace("\n", " ").strip())[0][0].split('__')[-1]