Text cleaning: scope of the package

[linogaliana]

Hi !

Thanks for this game changer package.

I am starting to work on a text classification issue.

Looking at the source code, I see some that some text cleaning routines, before performing any tokenization have been included.

First, they rely on NLTK corpus which can be challenging in offline environments as on premise Insee's infrastructure. However, my problem with that goes beyond that technical point.

In my opinion, this package should not be a swiss knife, this will make it harder to maintain (which, in itself is quite challenging even with small packages). Unless I am mistaken, I don't think fasttext performs that kind of cleaning before estimation or inference. I think it is a good idea: user needs to be aware of that methodological choice and do that by himself.

My proposal: getting rid of that kind of code. Users would provide ready to use text data for estimation or inference

Related issue

• Update unidecode issue by giving the choice to user via 'preprocess' parameter in predict torch-fastText#29 as well as Unidecode required for predict() torch-fastText#27 would be solved by that

[meilame-tayebjee]

Hello Lino,

Thanks for raising this issue!

The preprocess subpackage indeed includes a text cleaning routine based on NLTK and unidecode, specifically designed for Insee’s automatic coding framework and the French language. It can be very useful for some users and remains entirely optional.

For reference, the torchFastText class, along with FastTextModel and NGramTokenizer, is completely independent of NLTK or any text cleaning routine. Users can train and infer directly without preprocessing, and by default, the model does not apply any text processing (nor install NLTK or unidecode). Since PR InseeFrLab/torch-fastText#29, predictions only use preprocessing if explicitly enabled by the user.

That said, we acknowledge that including even an example of a cleaning routine can be debated - as it is an arbitrary and specific choice (out of almost infinite number of possibilities) while being a bit out of the scope of this package (which is mainly distributing the FastTextModel and a handy wrapper). Removing it is straightforward, but we should consider whether and where to provide it separately for users who may find it valuable.

[micedre]

I agree with @linogaliana, preprocessing depends too much on type of data for a generic code to be useful, while complexifying maintenance.

We could give some way to clean data in the exemple notebooks.

[meilame-tayebjee]

To be noted that the explainability module relies heavily on clean_text_feature from the preprocess subpackage.

I think it would be quite out of scope to adapt the explainability module for a custom preprocess function.

So two choices:

1. keep as is, with smooth imports and optional installations (default being models and tokenizer submodules)
2. completely remove both preprocess and explainability

Even though we lose a nice feature, the second choice seems the most reasonable: it centers the package around its main goal of distributing the fastText tokenizer and the PyTorch model.

[linogaliana]

I have not checked the explainability module but I think this one could be useful.

Isn't it possible to assume it takes as an input an object with a given class that embed a few methods ? (I think out loud).

Let's say it assumes an object trainingdata of class PyTorchFastText.data. If that objects as a method preprocess it will be applied. However, if is a standard dataframe (e.g. pd.DataFrame), then it returns an error and points to the documentation of the package where the creation of that class is explained.