Cons and Pros compared to data-juicer

[vincent-pli]

Guys, have you checked the data-juicer(https://github.com/modelscope/data-juicer), seems the propose is almost same, but simpler

[agoyal26]

Thanks for pointing it out @vincent-pli . We will check it out and get back
@shivdeep-singh-ibm @touma-I @Bytes-Explorer @shahrokhDaijavad

[shahrokhDaijavad]

@vincent-pli I will show the result of our investigation of data-juicer in one week, but for now, a few observations:

1. As for modality, they are ahead of us in multi-modal OPs (their terminology for transforms), but they don't have code modality at all. We are busy adding multi-modal transforms to catch up on that front.
2. For scalability, they use Ray/Slurm (New/Old technology). We use Ray/Spark (New/New technology).
3. They don't have "automation" of running transforms like what we have with KFP
4. Their development guide does not address developing new "scalable" OPs
5. As you have observed, data-juicer is simpler to use than DPK. We are working with a high priority on simplification of DPK.

[vincent-pli]

I agree with you, have a few additions to make：

1. data-juicer have two different concepts: Ops and Toolss , Ops can be orchestrated as pipeline(called recipe) and Tools cannot, just solo used pre/post the pipeline.
2. The "simplicity" I mentioned goes beyond just the usage methods; it also refers to the architecture/design aspect. the arch/design of data-juicer is very simple or sometime I think it's crude.
3. The simple design make a new ops/tools very easy

[shahrokhDaijavad]

Thanks for the additional comments, @vincent-pli.