Open
Conversation
Add defensive code to handle the case where state_queue.lock or job_queue.lock exist as directories instead of files. This situation has been observed in CI tests, though the root cause is unclear. The fix: - Changes the check from os.path.isfile() to os.path.exists() - Handles three cases: file (expected), directory (unexpected), other - Uses shutil.rmtree() to remove directories - Adds warning logging when a directory is found to aid debugging - Improves error handling with specific log messages This allows the minion to recover from corrupted lock state and continue operating, while logging information to help identify the underlying cause.
bdrx312
reviewed
Feb 14, 2026
Comment on lines
+407
to
+421
| # Defensive check: Clean up corrupted lock files (e.g., directories from OverlayFS races) | ||
| if os.path.exists(lock_fn) and not os.path.isfile(lock_fn): | ||
| _raise_error(f"lock_fn {lock_fn} exists and is not a file") | ||
| log.warning( | ||
| "Lock %s exists but is not a file (%s), cleaning up. " | ||
| "This may indicate an OverlayFS race or filesystem issue.", | ||
| lock_fn, | ||
| "directory" if os.path.isdir(lock_fn) else "other", | ||
| ) | ||
| try: | ||
| if os.path.isdir(lock_fn): | ||
| shutil.rmtree(lock_fn) | ||
| else: | ||
| os.remove(lock_fn) | ||
| except OSError as exc: | ||
| _raise_error(f"Failed to clean up corrupted lock {lock_fn}: {exc}") |
Contributor
There was a problem hiding this comment.
If you leave this in after troubleshooting then I suggest refactoring into a function since this appears to be exactly the same as the code added above at 325-339
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
What issues does this PR fix or reference?
Fixes
Previous Behavior
Remove this section if not relevant
New Behavior
Remove this section if not relevant
Merge requirements satisfied?
[NOTICE] Bug fixes or features added to Salt require tests.
Commits signed with GPG?
Yes/No