-
Notifications
You must be signed in to change notification settings - Fork 62
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
956da9a
commit 1e08973
Showing
9 changed files
with
1,019 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
|
||
# Contributing | ||
|
||
Contributions are welcome! Whether you have ideas for improving the internal workings of TwinTrim, such as optimizing performance or refining algorithms, or you want to enhance the user interface of the CLI tool for a better user experience, your input is valuable. Please fork the repository and submit a pull request with your improvements or new features. | ||
|
||
Please refer to the [CONTRIBUTING GUIDELINES](../CONTRIBUTION_GUIDELINES.md) for guidelines on how to contribute. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
# Features | ||
|
||
- **Duplicate Detection**: Scans directories to detect duplicate files based on file content rather than just filenames. | ||
- **Automatic or Manual Removal**: Choose to handle duplicates automatically using the `--all` flag or manually select which files to delete. | ||
- **Customizable Filters**: Set filters for minimum and maximum file sizes, file types, and specific filenames to exclude from the scan. | ||
- **Multi-Threaded Processing**: Utilizes multi-threading to quickly scan and process large numbers of files concurrently. | ||
- **Deadlock Prevention**: Implements locks to prevent deadlocks during multi-threaded operations, ensuring smooth and safe execution. | ||
- **User-Friendly Interface**: Offers clear prompts and feedback via the command line, making the process straightforward and interactive. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
# How It Works | ||
|
||
## Core Components | ||
|
||
1. **File Metadata Management**: | ||
- Uses `AllFileMetadata` and `FileMetadata` classes to manage file information, such as modification time and file paths. | ||
- Maintains metadata in two dictionaries (`store` and `normalStore`) for handling different levels of duplicate management. | ||
|
||
2. **File Hashing**: | ||
- Generates a unique hash for each file using MD5 to identify duplicates by content. | ||
|
||
3. **File Filtering**: | ||
- The `FileFilter` class provides functionality to filter files based on size, type, and exclusions. | ||
|
||
4. **Duplicate Handling**: | ||
- Duplicate files are identified by comparing their hashes. | ||
- Based on file modification time, the latest file is retained, and older duplicates are removed. | ||
|
||
5. **Deadlock Prevention**: | ||
- Uses locks within multi-threaded processes to ensure that resources are accessed safely, preventing deadlocks that could otherwise halt execution. | ||
|
||
## Key Functions | ||
|
||
- **add_or_update_file**: Adds new files to the metadata store or updates existing entries if a duplicate is detected. | ||
- **add_or_update_normal_file**: Similar to `add_or_update_file` but manages duplicates in a separate store. | ||
- **handleAllFlag**: Handles duplicate removal automatically without user intervention. | ||
- **find_duplicates**: Finds duplicate files in the specified directory and prepares them for user review or automatic handling. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
# License | ||
|
||
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,56 @@ | ||
## Usage | ||
|
||
### Command Line Interface | ||
|
||
Run the script using the following command: | ||
```bash | ||
python -m twinTrim.main <directory> [OPTIONS] | ||
``` | ||
|
||
### Options | ||
|
||
- `--all`: Automatically delete duplicates without asking for confirmation. | ||
- `--min-size`: Specify the minimum file size to include in the scan (e.g., `10kb`). | ||
- `--max-size`: Specify the maximum file size to include in the scan (e.g., `1gb`). | ||
- `--file-type`: Specify the file type to include (e.g., `.txt`, `.jpg`). | ||
- `--exclude`: Exclude specific files by name. | ||
- `--label-color`: Set the font color of the output label of the progress bar. | ||
- `--bar-color`: Set the color of the progress bar. | ||
|
||
### Examples | ||
|
||
1. **Automatic Duplicate Removal**: | ||
```bash | ||
python -m twinTrim.main /path/to/directory --all | ||
``` | ||
|
||
2. **Manual Review and Removal**: | ||
```bash | ||
python -m twinTrim.main /path/to/directory | ||
``` | ||
|
||
3. **Filtered Scan by File Size and Type**: | ||
```bash | ||
python -m twinTrim.main /path/to/directory --min-size "50kb" --max-size "500mb" --file-type "txt" | ||
``` | ||
|
||
## Dependencies | ||
|
||
- Python 3.6+ | ||
- `click` for command-line interaction | ||
- `tqdm` for progress bars | ||
- `concurrent.futures` for multi-threaded processing | ||
- `beaupy` for interactive selection | ||
|
||
## Installation | ||
|
||
Clone the repository and install the required dependencies using Poetry: | ||
|
||
```bash | ||
git clone https://github.com/Kota-Karthik/twinTrim.git | ||
cd twinTrim | ||
poetry install | ||
poetry shell | ||
``` | ||
|
||
If you haven't installed Poetry yet, you can do so by following the instructions on the [Poetry website](https://python-poetry.org/docs/#installation). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,99 @@ | ||
# Getting Started with TwinTrim | ||
|
||
Welcome to the TwinTrim getting started guide! This document will walk you through the steps to get started with the project. | ||
|
||
### Prerequisites | ||
|
||
Before you begin, ensure you have met the following requirements: | ||
|
||
- **Python Installation**: Make sure you have Python 3.6 or above installed on your system. You can download it from the [official Python website](https://www.python.org/downloads/). | ||
- **Git**: Ensure you have Git installed to clone the repository. You can get it from the [official Git website](https://git-scm.com/downloads). | ||
- **GitHub Account**: A GitHub account is required if you plan to contribute to the project. | ||
|
||
### Installing TwinTrim | ||
|
||
To install TwinTrim, follow these steps: | ||
|
||
1. **Clone the Repository**: Clone the repository from GitHub to your local machine. | ||
|
||
```bash | ||
git clone https://github.com/Kota-Karthik/twinTrim.git | ||
``` | ||
|
||
2. **Navigate to the Project Directory**: Change into the project directory. | ||
|
||
```bash | ||
cd twinTrim | ||
``` | ||
|
||
3. **Install Dependencies**: Install the required dependencies using the following command. If you are using `poetry`, you can install dependencies with: | ||
|
||
```bash | ||
poetry install | ||
``` | ||
|
||
Alternatively, if you are using `pip`, install the dependencies from `requirements.txt`: | ||
|
||
```bash | ||
pip install -r requirements.txt | ||
``` | ||
|
||
### Running TwinTrim | ||
|
||
To run TwinTrim, follow these steps: | ||
|
||
1. **Navigate to the Project Directory** (if not already there): | ||
|
||
```bash | ||
cd twinTrim | ||
``` | ||
|
||
2. **Run the Application**: Start the application with the following command: | ||
|
||
```bash | ||
python -m twinTrim.main <directory> [OPTIONS] | ||
``` | ||
|
||
Replace `<directory>` with the path to the directory you want to scan. You can add options such as `--all` to automatically remove duplicates or specify filters (e.g., `--min-size` or `--file-type`). | ||
|
||
The application will scan the directory and, based on your options, either prompt you to review the duplicates or automatically remove them. | ||
|
||
### Contributing to TwinTrim | ||
|
||
We welcome contributions! Follow these steps to contribute: | ||
|
||
1. **Fork the Repository**: Fork the repository on GitHub from [here](https://github.com/Kota-Karthik/twinTrim/fork). | ||
|
||
2. **Clone Your Fork**: Clone the forked repository to your local machine. | ||
|
||
```bash | ||
git clone https://github.com/<your-username>/twinTrim.git | ||
``` | ||
|
||
3. **Create a New Branch**: Create a new branch to work on your changes. | ||
|
||
```bash | ||
git checkout -b new-branch-name | ||
``` | ||
|
||
4. **Make Changes**: Make the necessary changes to the code. | ||
|
||
5. **Commit Changes**: Commit your changes with a descriptive message. | ||
|
||
```bash | ||
git commit -m "Descriptive commit message" | ||
``` | ||
|
||
6. **Push Changes**: Push your changes to your fork on GitHub. | ||
|
||
```bash | ||
git push origin new-branch-name | ||
``` | ||
|
||
7. **Create a Pull Request**: Go to the GitHub repository and create a pull request from your fork's branch to the main repository's `main` branch. | ||
|
||
Your changes will be reviewed, and if everything looks good, they will be merged into the main project. | ||
|
||
### Contact | ||
|
||
If you have any questions or need further assistance, feel free to reach out to [Karthik Kota](https://www.linkedin.com/in/kota-karthikk/). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
|
||
site_name: TwinTrim Docs | ||
site_url: https://twintrim.com/docs | ||
repo_url: https://github.com/Kota-Karthik/twinTrim | ||
repo_name: Kota-Karthik/twinTrim | ||
theme: | ||
name: material | ||
features: | ||
- navigation.tabs | ||
- navigation.sections | ||
- navigation.instant | ||
- navigation.instant.progress | ||
icon: | ||
repo: fontawesome/brands/github | ||
|
||
palette: | ||
scheme: slate | ||
|
||
nav: | ||
- Home: index.md | ||
- Usage: Usage.md | ||
- Contributing: Contributing.md | ||
- How It Works: How-It-Works.md | ||
- License: License.md |
Oops, something went wrong.