You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
NeMo Curator is a Python library that consists of a collection of scalable data-mining modules for curating natural language processing (NLP) data for training large language models (LLMs). The modules within NeMo Curator enable NLP researchers to mine high-quality text at scale from massive uncurated web corpora. For a demonstration of how each of the modules in NeMo Curator improves downstream performance, check out the [module ablation](#module-ablation).
4
-
5
-
NeMo Curator is built on [Dask](https://www.dask.org/) and [RAPIDS](https://developer.nvidia.com/rapids) to scale data curation and provide GPU acceleration. The Python interface provides easy methods to expand the functionality of your curation pipeline without worrying about how it will scale. More information can be found in the [usage section](#usage). There are many ways to integrate NeMo Curator in your pipeline. Check out the [installation instructions](#installation) for how to get started using it.
6
-
7
-
## Features
8
-
We currently support the following data-curation modules. For more details on each module, visit its documentation page in the [NeMo framework user guide](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/index.html).
9
-
-[Data download and text extraction](docs/user-guide/Download.rst)
10
-
- Default implementations of download and extraction of Common Crawl, Wikipedia, and ArXiv data
11
-
- Users can easily customize the download and extraction and extend to other datasets
12
-
-[Language identification and separation](docs/user-guide/LanguageIdentificationUnicodeFormatting.rst)
13
-
- Language identification with [fastText](https://fasttext.cc/docs/en/language-identification.html) and [pycld2](https://pypi.org/project/pycld2/)
14
-
-[Text reformatting and cleaning](docs/user-guide/LanguageIdentificationUnicodeFormatting.rst)
15
-
- Fix unicode decoding errors via [ftfy](https://ftfy.readthedocs.io/en/latest/)
- Our implementation follows the approach of [OpenAI GPT3](https://arxiv.org/pdf/2005.14165.pdf) and [Microsoft Turing NLG 530B](https://arxiv.org/abs/2201.11990)
24
-
-[Distributed data classification](docs/user-guide/DistributedDataClassification.rst)
25
-
- Multi-node multi-GPU classifier inference
26
-
- Allows for sophisticated domain and quality classification
27
-
- Flexible interface for extending to your own classifier network
28
-
-[Personal identifiable information (PII) redaction](docs/user-guide/PersonalIdentifiableInformationIdentificationAndRemoval.rst)
29
-
- Idenficiation tools for removing addresses, credit card numbers, social security numbers and more.
30
-
31
-
These modules are designed to be flexible and allow for reordering with few exceptions. The [NeMo Framework Launcher](https://github.com/NVIDIA/NeMo-Megatron-Launcher) includes prebuilt pipelines for you to start with and modify as needed.
32
-
33
-
## Learn More
3
+
NeMo Curator is a Python library specifically designed for scalable and efficient dataset preparation. It greatly accelerates data curation by leveraging GPUs with [Dask](https://www.dask.org/) and [RAPIDS](https://developer.nvidia.com/rapids), resulting in significant time savings. The library provides a customizable and modular interface, simplifying pipeline expansion and accelerating model convergence through the preparation of high-quality tokens.
4
+
5
+
At the core of the NeMo Curator is the `DocumentDataset` which serves as the the main dataset class. It acts as a straightforward wrapper around a Dask `DataFrame`. The Python library offers easy-to-use methods for expanding the functionality of your curation pipeline while eliminating scalability concerns.
6
+
7
+
## Key Features
8
+
9
+
NeMo Curator provides a collection of scalable data-mining modules. Some of the key features include:
10
+
11
+
[Data download and text extraction](docs/user-guide/Download.rst)
12
+
13
+
- Default implementations for downloading and extracting Common Crawl, Wikipedia, and ArXiv data
14
+
- Easily customize the download and extraction and extend to other datasets
15
+
16
+
[Language identification and separation](docs/user-guide/LanguageIdentificationUnicodeFormatting.rst)
17
+
18
+
- Language identification with [fastText](https://fasttext.cc/docs/en/language-identification.html) and [pycld2](https://pypi.org/project/pycld2/)
19
+
20
+
[Text reformatting and cleaning](docs/user-guide/LanguageIdentificationUnicodeFormatting.rst)
21
+
22
+
- Fix unicode decoding errors via [ftfy](https://ftfy.readthedocs.io/en/latest/)
- Our implementation follows the approach of [OpenAI GPT3](https://arxiv.org/pdf/2005.14165.pdf) and [Microsoft Turing NLG 530B](https://arxiv.org/abs/2201.11990)
37
+
38
+
[Distributed data classification](docs/user-guide/DistributedDataClassification.rst)
39
+
40
+
- Multi-node, multi-GPU classifier inference
41
+
- Provides sophisticated domain and quality classification
42
+
- Flexible interface for extending to your own classifier network
43
+
44
+
[Personal identifiable information (PII) redaction](docs/user-guide/PersonalIdentifiableInformationIdentificationAndRemoval.rst)
45
+
46
+
- Identification tools for removing addresses, credit card numbers, social security numbers, and more
47
+
48
+
These modules offer flexibility and permit reordering, with only a few exceptions. In addition, the [NeMo Framework Launcher](https://github.com/NVIDIA/NeMo-Megatron-Launcher) provides pre-built pipelines that can serve as a foundation for your customization use cases.
49
+
50
+
## Resources
51
+
34
52
-[Documentation](docs/)
35
53
-[Examples](examples/)
36
-
-[Module Ablation and Compute Performance](#module-ablation-and-compute-performance)
54
+
-[Tutorials](tutorials/)
37
55
38
-
## Installation
56
+
## Get Started
39
57
40
-
NeMo Curator currently requires Python 3.10 and the GPU accelerated modules require CUDA 12 or above installed in order to be used.
58
+
This section explains how to install NeMo Curator and use the Python library, Python modules, and CLI scripts. It also includes a list of tutorials to help you get started right away. Finally, this section explains how to use the NeMo Framework Launcher as an alternative method for interfacing with NeMo Curator.
41
59
42
-
NeMo Curator can be installed manually by cloning the repository and installing as follows -
60
+
## Requirements
43
61
44
-
For CPU only modules:
45
-
```
46
-
pip install .
47
-
```
62
+
Before installing NeMo Curator, ensure that the following requirements are met:
NeMo Curator is available in the [NeMo Framework Container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo). The NeMo Framework Container provides an end-to-end platform for development of custom generative AI models anywhere. The latest release of NeMo Curator comes preinstalled in the container.
NeMo Curator is available in the [NeMo Framework Container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo/tags). The latest release of NeMo Curator comes preinstalled in the container.
98
+
99
+
## Use the Python Library
100
+
101
+
The following snippet demonstrates how to create a small data curation pipeline that downloads and curates a small subset of the Common Crawl dataset.
NeMo Curator provides a collection of robust python modules that can be chained together to construct your entire data curation pipeline. These modules can be run on your local machine or in a distributed compute environment like SLURM with no modifications. NeMo Curator provides simple base classes that you can inherit from to create your own filters, document modifiers, and other extensions without needing to worry about how they scale. The [examples](examples/) directory contains a bunch of scripts showcasing each of these modules. The data curation section of the [NeMo framework user guide](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/index.html) provides in-depth documentation on how each of the modules work. If you need more information to modify the NeMo Curator for your usecase, the [implementation section](#implementation) provides a good starting point.
117
+
## Explore NeMo Curator Tutorials
118
+
119
+
To get started with NeMo Curator, you can follow the tutorials available here: [Tutorials]
120
+
(https://github.com/NVIDIA/NeMo-Curator/tree/main/tutorials). These tutorials include:
121
+
122
+
-`tinystories` which focuses on data curation for training from scratch.
123
+
-`peft-curation` which focuses on data curation for parameter-efficient fine-tuning use-cases.
124
+
125
+
## Access Python Modules
126
+
127
+
The Data Curation section of the [NeMo Framework User Guide](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/index.html) provides in-depth information about how the Python modules work. The [examples](examples/) directory in the GitHub repository provides scripts that showcase these modules.
77
128
78
-
###Scripts
129
+
##Use CLI Scripts
79
130
80
-
We provide CLI scripts to use as well in case those are more convienent. The scripts under`nemo_curator/scripts` map closely with each of the created python modules. Visit the [documentation](docs) for each of the python modules for more information about the scripts associated with it.
131
+
NeMo Curator also offers CLI scripts for you to use. The scripts in`nemo_curator/scripts` map closely to the supplied Python modules. Refer to the [NeMo Framework User Guide](https://docs.nvidia.com/nemo-framework/user-guide/latest/datacuration/index.html) for more information about the Python modules and scripts.
81
132
133
+
## Use NeMo Framework Launcher
82
134
83
-
### NeMo Framework Launcher
84
-
[NeMo Megatron Launcher](https://github.com/NVIDIA/NeMo-Megatron-Launcher) is another way to interface with NeMo Curator. The launcher allows
85
-
for easy parameter and cluster configuration and will automatically generate the SLURM batch scripts that wrap around the CLI scripts required to run your pipeline.
86
-
Note: This is not the only way to run NeMo Curator on SLURM. There are example scripts in [`examples/slurm`](examples/slurm/) for running NeMo Curator on SLURM without the launcher.
135
+
As an alternative method for interfacing with NeMo Curator, you can use the [NeMo Framework Launcher](https://github.com/NVIDIA/NeMo-Megatron-Launcher). The launcher enables you to easily configure the parameters and cluster. It can also automatically generate the SLURM batch scripts that wrap around the CLI scripts required to run your pipeline.
136
+
137
+
In addition, other methods are available to run NeMo Curator on SLURM. For example, refer to the example scripts in [`examples/slurm`](examples/slurm/) for information on how to run NeMo Curator on SLURM without the NeMo Framework Launcher.
87
138
88
139
## Module Ablation and Compute Performance
89
140
90
-
The modules within NeMo Curator were in large part designed to curate high-quality documents from Common Crawl snapshots and to be able to do so
91
-
in a scalable manner. In order to assess the quality of the Common Crawl documents curated by the modules in NeMo Curator, we performed a series
92
-
of ablation experiments in which we trained a 357M-parameter GPT-style model on the datasets resulting from the different stages of our data curation
93
-
pipeline implemented in NeMo Curator. The figure below demonstrates that the different data curation modules implemented within NeMo Curator
94
-
lead to improved model zero-shot downstream task performance.
141
+
The modules within NeMo Curator were primarily designed to curate high-quality documents from Common Crawl snapshots in a scalable manner. To evaluate the quality of the curated Common Crawl documents, we conducted a series of ablation experiments. In these experiments, we trained a 357M-parameter GPT-style model using datasets generated at various stages of our data curation pipeline, which was implemented in NeMo Curator.
142
+
143
+
The following figure shows that the use of different data curation modules implemented in NeMo Curator led to improved model zero-shot downstream task performance.
In terms of scalability and compute performance, using the RAPIDS + Dask fuzzy deduplication, we are able to deduplicate the 1.1 Trillion token Red Pajama dataset in 1.8 hours using 64 A100s.
149
+
In terms of scalability and compute performance, using the combination of RAPIDS and Dask fuzzy deduplication enabled us to deduplicate the 1.1 Trillion token Red Pajama dataset in 1.8 hours with 64 NVIDIA A100 Tensor Core GPUs.
150
+
151
+
Additionally, using the CPU-based modules, the following table shows the time required and resulting data size reduction for each processing step [Common Crawl snapshot from November/December of 2020](https://commoncrawl.org/2020/12/nov-dec-2020-crawl-archive-now-available/) using 30 CPU nodes (with hardware similar to the `c5.24xlarge`[Amazon AWS C5 instance](https://aws.amazon.com/ec2/instance-types/c5/)).
101
152
102
-
Additionally, using the CPU-based modules the table below shows the time required and resulting data size reduction of each step of processing the [Common Crawl snapshot from November/December of 2020](https://commoncrawl.org/2020/12/nov-dec-2020-crawl-archive-now-available/) using 30 CPU nodes (with hardware similar to the `c5.24xlarge`[Amazon AWS C5 instance](https://aws.amazon.com/ec2/instance-types/c5/)):
103
153
104
154
<table>
105
155
<thead>
@@ -132,8 +182,9 @@ Additionally, using the CPU-based modules the table below shows the time require
132
182
</tbody>
133
183
</table>
134
184
135
-
## Implementation
136
185
137
-
As mentioned above, the modules within NeMo Curator enable users to scale data-mining and NLP processing tasks to many nodes within a compute cluster.
138
-
The modules accomplish this using [Dask](https://www.dask.org/) with [cuDF](https://docs.rapids.ai/api/cudf/nightly/user_guide/10min/) (for the GPU-accelerated modules).
139
-
At the core of the NeMo Curator, `DocumentDataset` (the main dataset class) is just a simple wrapper around a Dask dataframe. Dask allows NeMo Curator to scale to arbitrary cluster sizes, and it supports a variety of distributed computing platforms. It supports reading and writing to different file formats, and it can balance these operations among nodes in the cluster. Importantly, Dask also supports the RAPIDS cuDF library for GPU-acclerated exact and fuzzy deduplication.
186
+
## Contribute to NeMo Curator
187
+
188
+
We welcome community contributions! Please refer to [CONTRIBUTING.md](https://github.com/NVIDIA/NeMo/blob/stable/CONTRIBUTING.md) for the process.
189
+
190
+
To contribute an article to the collection, please submit a pull request to the ``gh-pages-src`` branch of this repository. For detailed information, please consult the README located at the [gh-pages-src branch](https://github.com/NVIDIA/NeMo/tree/gh-pages-src#readme).
0 commit comments