Releases: Particle1904/DatasetHelpers
v2.9.0
Dataset Processor Tools is a comprehensive set of tools designed for processing image datasets for machine learning. With these tools, you can easily discard low-resolution images, resize images while preserving their aspect ratio, generate tags using a pre-trained model, mass edit .txt files with tags, and even manually edit .txt files with ease.
The download comes bundled with all the necessary dependencies, including OnnxRuntime .dll/.so files, to ensure a seamless experience. Please note that some of these files can be quite large due to their importance in the functionality of the tools.
For detailed instructions and examples, be sure to visit the Wiki page. You'll find comprehensive guidance on how to make the most of Dataset Processor Tools.
New Features:
- Adds experimental support for WD Large Auto Tagger:
- Larger auto tagger model with more data and more tags.
- I'm looking into quantizing the model for a future version to reduce its size.
Developer notes: I wasn't able to run it on my integrated GPU since the model requires a lot of VRAM when running with GPU and DirectML and for some reason I can't get the C# OnnxRuntime to work with GPU in 64bits builds. I was able to run it in CPU just fine though, so please let me know if you found any issues with it.
Requirements:
This software requires the following runtimes or newest versions, so be sure to install them:
.NET Desktop Runtime 8.0.2 or newer
For Windows users: Visual C++ Redistributable for Visual Studio 2019 for running the Model
For MAC users please check this Issue on how to build yourself, don't forget to read my comment!
v2.8.3
Dataset Processor Tools is a comprehensive set of tools designed for processing image datasets for machine learning. With these tools, you can easily discard low-resolution images, resize images while preserving their aspect ratio, generate tags using a pre-trained model, mass edit .txt files with tags, and even manually edit .txt files with ease.
The download comes bundled with all the necessary dependencies, including OnnxRuntime .dll/.so files, to ensure a seamless experience. Please note that some of these files can be quite large due to their importance in the functionality of the tools.
For detailed instructions and examples, be sure to visit the Wiki page. You'll find comprehensive guidance on how to make the most of Dataset Processor Tools.
Fixes
Fix for the OnnxRuntime Extensions crash in Linux builds.
Requirements:
This software requires the following runtimes or newest versions, so be sure to install them:
.NET Desktop Runtime 8.0.2 or newer
For Windows users: Visual C++ Redistributable for Visual Studio 2019 for running the Model
For MAC users please check this Issue on how to build yourself, don't forget to read my comment!
v2.8.2
Dataset Processor Tools is a comprehensive set of tools designed for processing image datasets for machine learning. With these tools, you can easily discard low-resolution images, resize images while preserving their aspect ratio, generate tags using a pre-trained model, mass edit .txt files with tags, and even manually edit .txt files with ease.
The download comes bundled with all the necessary dependencies, including OnnxRuntime .dll/.so files, to ensure a seamless experience. Please note that some of these files can be quite large due to their importance in the functionality of the tools.
For detailed instructions and examples, be sure to visit the Wiki page. You'll find comprehensive guidance on how to make the most of Dataset Processor Tools.
Fixes
Fix for the missing OnnxRuntime files in Linux builds.
Requirements:
This software requires the following runtimes or newest versions, so be sure to install them:
.NET Desktop Runtime 8.0.2 or newer
For Windows users: Visual C++ Redistributable for Visual Studio 2019 for running the Model
For MAC users please check this Issue on how to build yourself, don't forget to read my comment!
v2.8.1
Dataset Processor Tools is a comprehensive set of tools designed for processing image datasets for machine learning. With these tools, you can easily discard low-resolution images, resize images while preserving their aspect ratio, generate tags using a pre-trained model, mass edit .txt files with tags, and even manually edit .txt files with ease.
The download comes bundled with all the necessary dependencies, including OnnxRuntime .dll/.so files, to ensure a seamless experience. Please note that some of these files can be quite large due to their importance in the functionality of the tools.
For detailed instructions and examples, be sure to visit the Wiki page. You'll find comprehensive guidance on how to make the most of Dataset Processor Tools.
Fixes
Hotfixes to the Gallery and Inpaint pages; Mask images are now placed inside a new folder.
Requirements:
This software requires the following runtimes or newest versions, so be sure to install them:
.NET Desktop Runtime 8.0.2 or newer
For Windows users: Visual C++ Redistributable for Visual Studio 2019 for running the Model
For MAC users please check this Issue on how to build yourself, don't forget to read my comment!
v2.8.0
Dataset Processor Tools is a comprehensive set of tools designed for processing image datasets for machine learning. With these tools, you can easily discard low-resolution images, resize images while preserving their aspect ratio, generate tags using a pre-trained model, mass edit .txt files with tags, and even manually edit .txt files with ease.
The download comes bundled with all the necessary dependencies, including OnnxRuntime .dll/.so files, to ensure a seamless experience. Please note that some of these files can be quite large due to their importance in the functionality of the tools.
For detailed instructions and examples, be sure to visit the Wiki page. You'll find comprehensive guidance on how to make the most of Dataset Processor Tools.
New Features:
IMPORTANT: Due to a small bug in this version, please manually download the cliptokenizer.onnx file as described in this issue and place it inside the models folder. Issue is solved in the next version.
- AI-based Inpainting:
- Added support for inpainting images using the LaMa AI model. This feature can clean text, watermarks, logos, or objects from images.
- Uses a tile-based approach, chopping the original image and its mask into 512x512 tiles to avoid unnecessary inpainting and optimize resource use.
- Supports both manual mask creation using a masking tool (Left-click to paint white, Right-click to paint black) and pre-existing masks named in the format: "<filename>_mask.jpeg" (e.g., "1.png" should have a mask named "1_mask.jpeg").
- Note: Inpainting doesn't work with DirectML-GPU at this moment in time, seems to be a limitation in the LaMa.onnx model since its using custom pytorch operations for Fourier Convolutions.
- Settings Management:
- Entire settings page reworked to use .json files instead of .cfg files.
- Each page in the app now has its own configurable settings set.
- Upscaling Models:
- Added six more AI upscaling models.
- Manual Crop Page:
- Added a button to copy the currently selected image to the output folder when cropping isn't necessary.
- Editor Page Navigation Improvements:
- Editor control now automatically focuses after navigating between images.
- Caret position in the Editor retains its last character-based position or snaps to the end when navigating between images.
- Note: While Alt + Left/Right arrow keys still function for navigation, they may occasionally produce alt codes. F1 and F2 or Mouse 4 and Mouse 5 are recommended for navigation.
Requirements:
This software requires the following runtimes or newest versions, so be sure to install them:
.NET Desktop Runtime 8.0.2 or newer
For Windows users: Visual C++ Redistributable for Visual Studio 2019 for running the Model
For MAC users please check this Issue on how to build yourself, don't forget to read my comment!
v2.7.0
Dataset Processor Tools is a comprehensive set of tools designed for processing image datasets for machine learning. With these tools, you can easily discard low-resolution images, resize images while preserving their aspect ratio, generate tags using a pre-trained model, mass edit .txt files with tags, and even manually edit .txt files with ease.
The download comes bundled with all the necessary dependencies, including OnnxRuntime .dll/.so files, to ensure a seamless experience. Please note that some of these files can be quite large due to their importance in the functionality of the tools.
For detailed instructions and examples, be sure to visit the Wiki page. You'll find comprehensive guidance on how to make the most of Dataset Processor Tools.
New Features:
- Upscaling Image Setup:
- Exposes the Upscaling feature to the GUI.
- 19 Upscaler Models available for 2x and 4x upscaling. Most of them come from this repository.
Fixes
- Fix a bug introduced in the latest version where it would fail to load any AI model.
Requirements:
This software requires the following runtimes or newest versions, so be sure to install them:
.NET Desktop Runtime 8.0.2 or newer
For Windows users: Visual C++ Redistributable for Visual Studio 2019 for running the Model
For MAC users please check this Issue on how to build yourself, don't forget to read my comment!
v2.6.0 - AI Backend Update
Dataset Processor Tools is a comprehensive set of tools designed for processing image datasets for machine learning. With these tools, you can easily discard low-resolution images, resize images while preserving their aspect ratio, generate tags using a pre-trained model, mass edit .txt files with tags, and even manually edit .txt files with ease.
The download comes bundled with all the necessary dependencies, including OnnxRuntime .dll/.so files, to ensure a seamless experience. Please note that some of these files can be quite large due to their importance in the functionality of the tools.
For detailed instructions and examples, be sure to visit the Wiki page. You'll find comprehensive guidance on how to make the most of Dataset Processor Tools.
New Features:
- Migration to OnnxRuntime:
- The entire AI Backend has been revamped to utilize OnnxRuntime directly, replacing the dependency on ML.NET. This transition enables the flexibility to switch between providers for AI model execution and training.
- Extensive Provider Support: OnnxRuntime boasts support for 11 diverse providers, including CUDA, ROCm, DirectML, and others. Consequently, models in Onnx format can theoretically operate across a broad spectrum of devices. Testing on my Integrated AMD GPU demonstrated successful execution of all models via DirectML in x86 builds.
- Enhanced Model Loading: On model loading, OnnxRuntime now sequentially attempts loading via DirectML -> CUDA -> ROCm before resorting to CPU fallback.
- GPU Inference Limitation: A drawback observed entails GPU inference functioning solely in x86 builds, prompting the creation of an issue in the OnnxRuntime repository for further investigation. Despite attempts, GPU inference remains unresolved in x64 builds.
- Upscaling Image Setup:
- The groundwork for upscaling images using AI models has been laid, although this feature is currently not accessible via the GUI. Future plans include integrating a dedicated page to upscale all images within a folder and optionally upscale small images during dataset operations like resizing or content-aware cropping.
Requirements:
This software requires the following runtimes or newest versions, so be sure to install them:
.NET Desktop Runtime 8.0.2 or newer
For Windows users: Visual C++ Redistributable for Visual Studio 2019 for running the Model
For MAC users please check this Issue on how to build yourself, don't forget to read my comment!
v2.5.0
Dataset Processor Tools is a comprehensive set of tools designed for processing image datasets for machine learning. With these tools, you can easily discard low-resolution images, resize images while preserving their aspect ratio, generate tags using a pre-trained model, mass edit .txt files with tags, and even manually edit .txt files with ease.
The download comes bundled with all the necessary dependencies, including OnnxRuntime .dll/.so files, to ensure a seamless experience. Please note that some of these files can be quite large due to their importance in the functionality of the tools.
For detailed instructions and examples, be sure to visit the Wiki page. You'll find comprehensive guidance on how to make the most of Dataset Processor Tools.
New Features:
- Support for .AVIF images:
- Now you can process .AVIF images seamlessly along with other supported formats.
- Cancel Button:
- Added a Cancel button in relevant pages so users can finally cancel long-running operations instead of having to restart the app.
- Token Counting Feature:
- Added a feature to count the Tokens in the Editor Page using a custom Onnx operation to tokenize text for CLIP models. Stable Diffusion models use 75 tokens by default (even though workarounds exist in Trainers).
Fixes:
- Resizing and Cropping Operations:
- Fixed resizing and cropping operations to properly fill empty backgrounds with white.
- Gallery Page Improvements:
- Various fixes in the Gallery page like properly downscaling images when reducing the display size, added pagination to support as many images as the user has.
- Improved Manual Crop Page:
- Improved how the Manual Crop page displays images. It can now support any image size, it will downscale to display but will still use the original size when cropping.
Requirements:
This software requires the following runtimes or newest versions, so be sure to install them:
.NET Desktop Runtime 8.0.2 or newer
For Windows users: Visual C++ Redistributable for Visual Studio 2019 for running the Model
For MAC users please check this Issue on how to build yourself, don't forget to read my comment!
v2.4.1
Dataset Processor Tools is a comprehensive set of tools designed for processing image datasets for machine learning. With these tools, you can easily discard low-resolution images, resize images while preserving their aspect ratio, generate tags using a pre-trained model, mass edit .txt files with tags, and even manually edit .txt files with ease.
The download comes bundled with all the necessary dependencies, including OnnxRuntime .dll/.so files, to ensure a seamless experience. Please note that some of these files can be quite large due to their importance in the functionality of the tools.
For detailed instructions and examples, be sure to visit the Wiki page. You'll find comprehensive guidance on how to make the most of Dataset Processor Tools.
Fixes:
- Security Enhancements: Upgraded dependencies, addressing vulnerabilities like those identified in ImageSharp.
- Enhanced User Experience: Yolo model no longer appears erroneously as a generator model on the Generation Page.
- Performance Optimization: Implemented a temporary solution for downsizing oversized images in the Manual Crop page.
Requirements:
This software requires the following runtimes or newest versions, so be sure to install them:
.NET Desktop Runtime 8.0.2 or newer
For Windows users: Visual C++ Redistributable for Visual Studio 2019 for running the Model
For MAC users please be sure to check this Issue on how to build yourself, don't forget to read my comment!
v2.4.0
Dataset Processor Tools is a comprehensive set of tools designed for processing image datasets for machine learning. With these tools, you can easily discard low-resolution images, resize images while preserving their aspect ratio, generate tags using a pre-trained model, mass edit .txt files with tags, and even manually edit .txt files with ease.
The download comes bundled with all the necessary dependencies, including OnnxRuntime .dll/.so files, to ensure a seamless experience. Please note that some of these files can be quite large due to their importance in the functionality of the tools.
For detailed instructions and examples, be sure to visit the Wiki page. You'll find comprehensive guidance on how to make the most of Dataset Processor Tools.
New Features:
- WD Auto Tagger v3 Support:
- Added support for generating tags with the newly released WD Auto Tagger v3 model by SmilingWolf. Check his HuggingFace page for more information.
- As always, it will be automatically downloaded if you try to use it for the first time. I had to edit the input and output layer names to integrate it.
- Not sure on optimal threshold values for it yet.
- Gallery Page for Quick Image Selection:
- Introducing a Gallery page for swiftly selecting images within a dataset for deletion. Currently supports datasets with up to 2500 images. For larger datasets, splitting into multiple folders is recommended.
- Still WIP! Needs a better solution to deal with larger datasets that doesn't involve limiting total number of images.
- Improved Log Messages:
- Logs presented to the user are now color-coded, providing a clearer distinction between informational, warning, and error logs.
Fixes:
- Resolved an issue where the application incorrectly interpreted keyboard inputs on pages where it shouldn't. Keyboard shortcuts are now exclusive to the Editor Page, ensuring faster iteration on datasets.
Requirements:
This software requires the following runtimes or newest versions, so be sure to install them:
.NET Desktop Runtime 8.0.2 or newer
For Windows users: Visual C++ Redistributable for Visual Studio 2019 for running the Model
For MAC users please be sure to check this Issue on how to build yourself, don't forget to read my comment!