Skip to content

Commit 7f7514c

Browse files
authored
#3 Implement EmbeddingToTextPipeline and TextToEmbedding Pipeline (#35)
* Add text pipeline * Build sentences one by one * Create a separate preprocessing pipeline for text distribute responsibility across pipelines * Improve decoding and encoding to parallelize by keeping GPU busy, this is done by encoding and decoding multiple texts at a time * Make pipelines normal classes rather than data classes as it doesnt manage any data * Move TexToEmbedding config params from abstract * Update docs for TextToEmbedding overwrites * Improve docs for text pipelines * Add interface documentation to docs * Add unit tests for text pipelines * Add optional dependencies for huggingface using tag * Remove commas between configs * Improve docs for text pipelines * It should be list * Move text segmentation pipeline to text file * Fix linting issues * Ignore spacy types * Fix black linter issues * Ignore spacy types * Reformat text file with black * Use copyless np arrays * Add more robust tests * Fix linting issues * add more tests * Fix linting issues * Ignore spacy types * Ignore spacy types * Add new changes * Improve text pipeline * Add more robust tests * Fix linting issues * Fix linting issues * Fix linting issues * Fix linting issues * Fix mypy inheritance issue
1 parent dbdcb7c commit 7f7514c

File tree

4 files changed

+768
-3
lines changed

4 files changed

+768
-3
lines changed

huggingface_pipelines/__init__.py

Whitespace-only changes.

0 commit comments

Comments
 (0)