Speed up hook diff processing #27

oleander · 2025-02-07T21:56:05Z

No description provided.

Optimize PatchDiff with parallel processing Add rayon dependency to Cargo.toml Remove redundant patch tests Update Cargo.lock with new dependencies Remove profiling tests

feat: Add Finetune functionality and related files - Introduce finetune.rs to manage fine-tuning workflows with OpenAI. - Create finetune.md for documenting the finetuning process. - Update Cargo.toml and Cargo.lock with necessary dependencies for finetuning. - Add stats.json to track various parameters in the finetuning process. ```

Copilot reviewed 7 out of 14 changed files in this pull request and generated no comments.

Files not reviewed (7)

Cargo.toml: Evaluated as low risk
src/filesystem.rs: Evaluated as low risk
src/hook.rs: Evaluated as low risk
src/lib.rs: Evaluated as low risk
src/reinstall.rs: Evaluated as low risk
src/commit.rs: Evaluated as low risk
src/model.rs: Evaluated as low risk

Comments suppressed due to low confidence (5)

src/config.rs:14

[nitpick] The constant DEFAULT_MAX_TOKENS is set to 2024, which seems like a typo. It should probably be 2048 which is a more common default value for token limits.

const DEFAULT_MAX_TOKENS: i64 = 2024;

src/config.rs:49

[nitpick] The error message "Failed to create config directory at" could be improved to provide more context, such as suggesting checking directory permissions.

std::fs::create_dir_all(&self.dir).with_context(|| format!("Failed to create config directory at {:?}", self.dir))?;

src/config.rs:52

[nitpick] The error message "Failed to create config file at" could be improved to provide more context, such as suggesting checking file permissions.

File::create(&self.file).with_context(|| format!("Failed to create config file at {:?}", self.file))?;

src/install.rs:10

[nitpick] The error message should be more informative. Consider changing it to: "Hook already exists at {}. If you want to reinstall, please run 'git ai hook reinstall'. If you want to keep the existing hook, no further action is needed."

bail!("Hook already exists at {}, please run 'git ai hook reinstall'", hook_file);

src/openai.rs:55

The variable name 'choise' is misspelled. It should be renamed to 'choice'.

let choise = chat

Add instruction template constant to commit.rs - Introduce `INSTRUCTION_TEMPLATE` constant for prompt file content. ```

- Remove the import of `Model` and update the `available_tokens` calculations in the `call` function.

- Change maximum commit length from 72 to {{max_commit_length}} characters.

This reverts commit 7b9aa2f.

Update Command enum definition Based only on the changes visible in the diff, this commit modifies the existing Command enum without altering its structure or commands. ```

Based only on the changes visible in the diff, this commit removes the stats.json file.

Remove install, reinstall, and uninstall modules Based only on the changes visible in the diff, this commit deletes the files src/install.rs, src/reinstall.rs, and src/uninstall.rs. ```

Based only on the changes visible in the diff, this commit modifies the default model name in the Args implementation from "gpt-4o" to "gpt-4o-mini".

Create hook stress test script Based only on the changes visible in the diff, this commit adds a new script for testing various operations in a Git hook context. ```

Add comprehensive tests script Based only on the changes visible in the diff, this commit adds a new Fish script `comprehensive-tests` that includes a series of tests for various Git operations. ```

- Update file mode from 644 to 755.

…m `.env.local` This commit updates the `comprehensive-tests` script by adding logic to read and load environment variables from a

…mpt.md'

Based only on

Based on the changes visible in the diff, this commit updates the comprehensive testing scripts, modifies several source files related to- 'hook.rs','patch_test.rs', and adds new methods and tests. Here is a summary of specific changes: - In `comprehensive-tests` script, add `--debug` flag to `cargo install` command - Adjust logic to check for ample remaining tokens in `hook.rs` - Modify Diff and PatchRepository implementations for `hook.rs` and `patch_test.rs`. - Modify test cases in the `patch_test.rs` script. Please note each individual file has significant changes that add, modify or enhance the functionality as per the needs reflected in the diffs. However, all changes stick to the theme of improving handling of patches, diffs, and commit-related functionalities.

Based only on the changes visible in the diff, this commit: - Adds nine lines to check for an empty diff and handle amend operations in `src/bin/hook.rs` - Removes four lines related to the error message for no changes found to commit - Modifies a line to ensure a minimum of 512 remaining tokens - Appends a four-line snippet to handle amend operations when the source is a commit

Based only on the changes visible in the diff, this commit: - Replaces the commented clean-up command in comprehensive-tests script with an active one.

This commit introduces the initial boilerplate for the `hook.rs` file. The added codes allow for dead code, which is often used during the early stages of development. This single-line addition simply contributes to the initial setup of the file.

Based only on the changes visible in the diff, this commit: - Adds code for checking if a commit message already exists and is not empty in the file hook.rs, - If the message exists, a debug log message is displayed, and the function returns, clearing the progress bar.

…PatchRepository` trait Based only on the changes visible in the diff, this commit: - Adds the `to_commit_diff` method to the `PatchRepository` trait which returns a `Result<git2::Diff<'_>>` - Implements `to_commit_diff` in the `PatchRepository` trait for `Repository` class, where it configures diff options and conducts the diff operation based on the provided tree option - Adds the `configure_commit_diff_options` method to the `PatchRepository` trait which doesn't return anything but changes the state of provided `DiffOptions` - Implements `configure_commit_diff_options` in the `PatchRepository` trait for `Repository` class, where it sets various options for a diff operation - Replaces the usage of `to_diff` method with `to_commit_diff` in the `PatchRepository` implementation for `Repository`.

The max_tokens_per_file calculation within the `process_chunk` function of hook.rs now considers the case of zero remaining files. If no files are remaining, the total_remaining value is assigned to max_tokens_per_file directly.

…t.rs

…ic in hook.rs

* test * Add profiling feature with macro for measuring execution time * Refactor diff processing, optimize token handling and storage * Update test signatures and add profiling tests * Refactor GitFile signatures and commit messages Optimize PatchDiff with parallel processing Add rayon dependency to Cargo.toml Remove redundant patch tests Update Cargo.lock with new dependencies Remove profiling tests * Update Cargo.lock dependencies and checksums * Update dependencies in Cargo.toml and Cargo.lock * Add StringPool for efficient memory use in PatchDiff * Update dependencies in Cargo.toml and Cargo.lock * Add `num_cpus` crate and parallelize file processing * Refactor file processing to use parallel chunks and atomic tokens * Remove redundant import of `bail` from anyhow * Sort files by token count in `PatchDiff` implementation. * Delete test.txt file * Improve error handling and path management in config and style modules * Add tests for StringPool functionality in hook.rs * Update default model and add profiling to model and commit functions * Add profiling to filesystem module functions * Implement token counting and generation for commit messages * Add documentation for Filesystem, File, and Dir structs in filesystem.rs * Refactor commit message generation methods and file handling logic * Implement configuration file management and update functions in App * Implement parallel processing of diff data in PatchDiff trait * ``` feat: Add Finetune functionality and related files - Introduce finetune.rs to manage fine-tuning workflows with OpenAI. - Create finetune.md for documenting the finetuning process. - Update Cargo.toml and Cargo.lock with necessary dependencies for finetuning. - Add stats.json to track various parameters in the finetuning process. ``` * ``` Add instruction template constant to commit.rs - Introduce `INSTRUCTION_TEMPLATE` constant for prompt file content. ``` * Remove unused import of `std::fs` from `commit.rs` file. * Remove unused import and adjust available tokens calculation - Remove the import of `Model` and update the `available_tokens` calculations in the `call` function. * Update max commit length in prompt guidelines - Change maximum commit length from 72 to {{max_commit_length}} characters. * ``` Modify imports and refactor filesystem profiling Based only on the changes visible in the diff, this commit: - Removes unnecessary profiling statements and imports in `filesystem.rs`. - Adds an import for `App` from `config` in `main.rs`. ``` * Add directory creation for hooks if it does not exist - Implement logic to check for the existence of the hooks directory and create it if it's missing. * Add dead code allowance in filesystem.rs Based only on the changes visible in the diff, this commit adds a line to allow dead code in the `filesystem.rs` file. * Revert "```" This reverts commit 7b9aa2f. * ``` Update Command enum definition Based only on the changes visible in the diff, this commit modifies the existing Command enum without altering its structure or commands. ``` * Delete stats.json file Based only on the changes visible in the diff, this commit removes the stats.json file. * ``` Remove install, reinstall, and uninstall modules Based only on the changes visible in the diff, this commit deletes the files src/install.rs, src/reinstall.rs, and src/uninstall.rs. ``` * Build inline * Update default model name in Args implementation Based only on the changes visible in the diff, this commit modifies the default model name in the Args implementation from "gpt-4o" to "gpt-4o-mini". * ``` Create hook stress test script Based only on the changes visible in the diff, this commit adds a new script for testing various operations in a Git hook context. ``` * ``` Add comprehensive tests script Based only on the changes visible in the diff, this commit adds a new Fish script `comprehensive-tests` that includes a series of tests for various Git operations. ``` * Change file permission of comprehensive-tests. - Update file mode from 644 to 755. * Update `comprehensive-tests` script to load environment variables from `.env.local` This commit updates the `comprehensive-tests` script by adding logic to read and load environment variables from a * Remove note about output being used as a git commit message from 'prompt.md' * Update comprehensive-tests script and prompt.md documentation Based only on * Update scripts and source code according to visible changes in the diff Based on the changes visible in the diff, this commit updates the comprehensive testing scripts, modifies several source files related to- 'hook.rs','patch_test.rs', and adds new methods and tests. Here is a summary of specific changes: - In `comprehensive-tests` script, add `--debug` flag to `cargo install` command - Adjust logic to check for ample remaining tokens in `hook.rs` - Modify Diff and PatchRepository implementations for `hook.rs` and `patch_test.rs`. - Modify test cases in the `patch_test.rs` script. Please note each individual file has significant changes that add, modify or enhance the functionality as per the needs reflected in the diffs. However, all changes stick to the theme of improving handling of patches, diffs, and commit-related functionalities. * Refactor `hook.rs` and ensure a minimum of 512 tokens Based only on the changes visible in the diff, this commit: - Adds nine lines to check for an empty diff and handle amend operations in `src/bin/hook.rs` - Removes four lines related to the error message for no changes found to commit - Modifies a line to ensure a minimum of 512 remaining tokens - Appends a four-line snippet to handle amend operations when the source is a commit * Update clean-up command in comprehensive-tests script Based only on the changes visible in the diff, this commit: - Replaces the commented clean-up command in comprehensive-tests script with an active one. * Add attribute to suppress dead code warnings in hook.rs * Add initial boilerplate for hook.rs This commit introduces the initial boilerplate for the `hook.rs` file. The added codes allow for dead code, which is often used during the early stages of development. This single-line addition simply contributes to the initial setup of the file. * Add debug message when a commit message already exists in hook.rs Based only on the changes visible in the diff, this commit: - Adds code for checking if a commit message already exists and is not empty in the file hook.rs, - If the message exists, a debug log message is displayed, and the function returns, clearing the progress bar. * Add `to_commit_diff` and `configure_commit_diff_options` methods to `PatchRepository` trait Based only on the changes visible in the diff, this commit: - Adds the `to_commit_diff` method to the `PatchRepository` trait which returns a `Result<git2::Diff<'_>>` - Implements `to_commit_diff` in the `PatchRepository` trait for `Repository` class, where it configures diff options and conducts the diff operation based on the provided tree option - Adds the `configure_commit_diff_options` method to the `PatchRepository` trait which doesn't return anything but changes the state of provided `DiffOptions` - Implements `configure_commit_diff_options` in the `PatchRepository` trait for `Repository` class, where it sets various options for a diff operation - Replaces the usage of `to_diff` method with `to_commit_diff` in the `PatchRepository` implementation for `Repository`. * Optimize max_tokens_per_file calculation in hook.rs The max_tokens_per_file calculation within the `process_chunk` function of hook.rs now considers the case of zero remaining files. If no files are remaining, the total_remaining value is assigned to max_tokens_per_file directly. * Refactor method calls and condition checks in openai.rs and patch_test.rs * Refine instructions and guidelines for generating git commit messages * Add error handling for raw SHA1 resolution in hook.rs * Refactor function calls in patch_test.rs and simplify conditional logic in hook.rs * Refactor reference resolution in hook.rs --------- Co-authored-by: Git AI Test <test@example.com>

oleander and others added 24 commits February 7, 2025 14:37

test

7d9843b

Add profiling feature with macro for measuring execution time

fe94188

Refactor diff processing, optimize token handling and storage

462b75d

Update test signatures and add profiling tests

2ea378d

Refactor GitFile signatures and commit messages

37e5b2e

Optimize PatchDiff with parallel processing Add rayon dependency to Cargo.toml Remove redundant patch tests Update Cargo.lock with new dependencies Remove profiling tests

Update Cargo.lock dependencies and checksums

c87b0d2

Update dependencies in Cargo.toml and Cargo.lock

7eb1eb7

Add StringPool for efficient memory use in PatchDiff

e2df5f4

Update dependencies in Cargo.toml and Cargo.lock

bf3110f

Add num_cpus crate and parallelize file processing

63a498c

Refactor file processing to use parallel chunks and atomic tokens

6a5051b

Remove redundant import of bail from anyhow

7ed087b

Sort files by token count in PatchDiff implementation.

51f9609

Delete test.txt file

5e50a25

Improve error handling and path management in config and style modules

d49f534

Add tests for StringPool functionality in hook.rs

600e5fd

Update default model and add profiling to model and commit functions

450381d

Add profiling to filesystem module functions

00faa02

Implement token counting and generation for commit messages

4a7d5d8

Add documentation for Filesystem, File, and Dir structs in filesystem.rs

a5833c8

Refactor commit message generation methods and file handling logic

5384690

Implement configuration file management and update functions in App

fa47b5b

Implement parallel processing of diff data in PatchDiff trait

c7778e6

oleander requested a review from Copilot February 8, 2025 05:11

Copilot AI reviewed Feb 8, 2025

View reviewed changes

Git AI Test added 4 commits February 8, 2025 06:13

```

1e56b0d

Add instruction template constant to commit.rs - Introduce `INSTRUCTION_TEMPLATE` constant for prompt file content. ```

Remove unused import of std::fs from commit.rs file.

1198750

Remove unused import and adjust available tokens calculation

140f2df

- Remove the import of `Model` and update the `available_tokens` calculations in the `call` function.

Update max commit length in prompt guidelines

e1f49e4

- Change maximum commit length from 72 to {{max_commit_length}} characters.

Git AI Test added 27 commits February 8, 2025 06:27

Revert "```"

aa4d073

This reverts commit 7b9aa2f.

```

6fd6ab8

Update Command enum definition Based only on the changes visible in the diff, this commit modifies the existing Command enum without altering its structure or commands. ```

Delete stats.json file

10192f6

Based only on the changes visible in the diff, this commit removes the stats.json file.

```

1e231fb

Remove install, reinstall, and uninstall modules Based only on the changes visible in the diff, this commit deletes the files src/install.rs, src/reinstall.rs, and src/uninstall.rs. ```

Build inline

42f27e2

Update default model name in Args implementation

b75147b

Based only on the changes visible in the diff, this commit modifies the default model name in the Args implementation from "gpt-4o" to "gpt-4o-mini".

```

a546bba

Create hook stress test script Based only on the changes visible in the diff, this commit adds a new script for testing various operations in a Git hook context. ```

```

596f662

Add comprehensive tests script Based only on the changes visible in the diff, this commit adds a new Fish script `comprehensive-tests` that includes a series of tests for various Git operations. ```

Change file permission of comprehensive-tests.

0f7af0c

- Update file mode from 644 to 755.

Update comprehensive-tests script to load environment variables fro…

5d5ce13

…m `.env.local` This commit updates the `comprehensive-tests` script by adding logic to read and load environment variables from a

Remove note about output being used as a git commit message from 'pro…

62e75f0

…mpt.md'

Update comprehensive-tests script and prompt.md documentation

4e8bbc9

Based only on

Update clean-up command in comprehensive-tests script

825150c

Based only on the changes visible in the diff, this commit: - Replaces the commented clean-up command in comprehensive-tests script with an active one.

Add attribute to suppress dead code warnings in hook.rs

3c4c51b

Add initial boilerplate for hook.rs

8f4942e

This commit introduces the initial boilerplate for the `hook.rs` file. The added codes allow for dead code, which is often used during the early stages of development. This single-line addition simply contributes to the initial setup of the file.

Optimize max_tokens_per_file calculation in hook.rs

d6efbfc

The max_tokens_per_file calculation within the `process_chunk` function of hook.rs now considers the case of zero remaining files. If no files are remaining, the total_remaining value is assigned to max_tokens_per_file directly.

Merge main into feature/speed, preserving performance improvements

207f2c3

Refactor method calls and condition checks in openai.rs and patch_tes…

b7cce70

…t.rs

Refine instructions and guidelines for generating git commit messages

9e568a3

Add error handling for raw SHA1 resolution in hook.rs

25b56cb

Merge remote-tracking branch 'origin/main' into feature/speed

944577b

Refactor function calls in patch_test.rs and simplify conditional log…

2fa2562

…ic in hook.rs

Refactor reference resolution in hook.rs

1bfd788

oleander merged commit 95b5977 into main Feb 8, 2025
5 checks passed

oleander deleted the feature/speed branch February 8, 2025 15:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Speed up hook diff processing #27

Speed up hook diff processing #27

oleander commented Feb 7, 2025

Speed up hook diff processing #27

Speed up hook diff processing #27

Conversation

oleander commented Feb 7, 2025

Choose a reason for hiding this comment