[Bug] Hunting - Add UTF-8 Encoding for all Read and Write Operations

[terrancedejesus]

Pull Request

Issue link(s):

• https://github.com/elastic/ia-trade-team/issues/280

Summary - What I changed

⚡ A list of hunting queries (see below) had ñ characters in the regex when normalizing paths in ES|QL queries. This was added intentionally to match on that character. However, when running hunting/generate_markdown.py script, we did not include UTF-8 encoding on the content when loading the TOML file, only when writing to Markdown. As a result @Aegrah pointed out that the ñ changed when writing to the Markdown files from a Windows host.

This is likely because UTF-8 was used on the writing operations and not the loading.

List of hunts affected:

• Low Occurrence of Suspicious Launch Agent or Launch Daemon
• Low Occurrence Rate of CreateRemoteThread by Source Process
• Low Occurrence of Drivers Loaded on Unique Hosts
• Persistence via Run Key with Low Occurrence Frequency
• Unique Windows Services Creation by Service File Name

Note that the hunts listed, both TOML and MD files, are correct in main and do not require further adjustment.

How To Test

1. From a Windows host, checkout the main
2. Run python ~/detection-rules/hunting/generate_markdown.py
3. Notice the diff on the queries listed above
4. Remove changes
5. Checkout bug-hunting-encoding-content-from-files branch
6. Run python ~/detection-rules/hunting/generate_markdown.py
7. Notice no changes

Checklist

• Added a label for the type of pr: bug, enhancement, Rule: New, Rule: Deprecation, Rule: Promote, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[protectionsmachine]

Bug - Guidelines

These guidelines serve as a reminder set of considerations when addressing a bug in the code.

Documentation and Context

• Provide detailed documentation (description, screenshots, reproducing the bug, etc.) of the bug if not already documented in an issue.
• Include additional context or details about the problem.
• Ensure the fix includes necessary updates to the release documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Code changes do not introduce new warnings or errors.
• Variables and functions are well-named and descriptive.
• Any unnecessary / commented-out code is removed.
• Ensure that the code is modular and reusable where applicable.
• Check for proper exception handling and messaging.

Testing

• New unit tests have been added to cover the bug fix or edge cases.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and detecting the bug fix (e.g., test logs, screenshots).
• Validate that any rules affected by the bug are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.

Additional Checks

• Ensure that the bug fix does not break existing functionality.
• Review the bug fix with a peer or team member for additional insights.
• Verify that the bug fix works across all relevant environments (e.g., different OS versions).
• Confirm that all dependencies are up-to-date and compatible with the changes.