Repositories list

• webdataset

  Public
  A high-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch.

  deep-learningpytorchdata-augmentation+ 2webdatasetwebdataset-format

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •198•2.5k•114•4•Updated Feb 12, 2025Feb 12, 2025

• WebDataset.jl

  Public
  A high performance I/O library for deep learning in Julia, based on the PyTorch WebDataset library

  deep-learningjuliashards+ 2tar-filesjulia-multithreading

  Julia

  •

  MIT License

  •1•10•0•50•Updated Nov 29, 2024Nov 29, 2024

• tarp

  Public
  Fast and simple stream processing of files in tar files, useful for deep learning, big data, and many other applications.

  Go

  •14•127•7•3•Updated Dec 10, 2023Dec 10, 2023

• wdsconversions

  Public
  conversions of standard pytorch datasets to webdataset format

  Jupyter Notebook

  •1•1•0•0•Updated Aug 20, 2022Aug 20, 2022