#1 Implement Dataset config for loading HuggingFace datasets #31
Conversation
facebook-github-bot
added
the
CLA Signed
|
what about adding some integration tests ? |
botirk38
changed the title
huggingface_pipelines/dataset.py
Outdated
| @dataclass | ||
| class TextDatasetConfig(DatasetConfig): | ||
| """ | ||
| Configuration for text datasets. | ||
|
|
||
| This class inherits from BaseDatasetConfig and can be used for | ||
| text-specific dataset configurations. | ||
| """ | ||
|
|
||
|
|
||
| @dataclass | ||
| class AudioDatasetConfig(DatasetConfig): |
There was a problem hiding this comment.
they should live in their corresponding modules, but let's move them once it's merged !
There was a problem hiding this comment.
Approved with some nits.
@botirk38 Please make sure to clean up linter / isort complaints before landing
| assert self.world_size >= 1, f"Invalid world_size: {self.world_size}. It should be >= 1." | ||
| assert 0 <= self.rank < self.world_size, f"Invalid rank: {self.rank}. It should be between 0 and {self.world_size - 1}." | ||
|
|
||
| def with_overwrites(self, overwrites: DatasetOverwrites): |
There was a problem hiding this comment.
since we use fairseq2, we can also just try fairseq2.utils.dataclass.update_dataclass here https://github.com/facebookresearch/fairseq2/blob/main/src/fairseq2/utils/dataclass.py#L36 (in combination with deepcopy())
| "pylint>=2.8.0", | ||
| ] | ||
|
|
||
| hg = ["transformers>=4.44.0", "datasets>=2.20.0"] |
There was a problem hiding this comment.
Do we need to constraint the version of transformers that much ?
huggingface_pipelines/dataset.py
Outdated
| Raises: | ||
| AssertionError: If world_size or rank are invalid. | ||
| """ | ||
| assert self.world_size >= 1, f"Invalid world_size: {self.world_size}. It should be >= 1." |
There was a problem hiding this comment.
nit comment, but this might help you in the future:
in standard SWE, assert is normally to assert low-level, internal variables that should act in certain way in normal cases, to define the points of fail fast in the program.
When it comes to validating user inputs, it is normally better to use "if" condition and raise ValueError
Why?
We need to implement this feature to standardize and simplify the process of loading and sharding Hugging Face datasets in SONAR pipelines. The primary use cases are:
This implementation will improve code maintainability, enhance reproducibility of experiments, and facilitate easier scaling of training processes across multiple GPUs or nodes.
How?
Key technical decisions made in this implementation:
DatasetConfig).DatasetOverwrites) to specify allowed configuration overwrites, ensuring type safety.datasetslibrary for actual dataset loading, leveraging its robust features.with_overwritesmethod for easy parameter overriding without modifying the original configuration.Test plan
To test these changes, we will: