-
Notifications
You must be signed in to change notification settings - Fork 7.1k
[ci] raydepsets relax operation: relax operation (3/3) #59910
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Elliot Barnwell <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Elliot Barnwell <elliot.barnwell@anyscale.com>
…ct/ray into elliot-barn/relax-operation-p1
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request introduces a new relax operation for raydepsets, which allows removing specified packages and their orphaned dependencies from a dependency set. The implementation is well-structured, breaking down the logic into a parser for lock files, a dependency graph manager, and the main CLI logic. The addition of unit tests for the new components and the relax operation is great.
I have a few suggestions to improve robustness and maintainability:
- In
cli.py, usingtempfile.TemporaryDirectorycontext manager for safer temporary file handling and avoiding mutation of input arguments. - A bug fix in
execute_depsetwhere an argument was missing in a method call. - Some improvements in the
parser.pyfor better memory efficiency and more robust parsing. - A minor consistency improvement in logging.
Overall, this is a solid contribution that adds valuable functionality.
| self.relax( | ||
| source_depset=depset.source_depset, | ||
| drop_packages=depset.drop_packages, | ||
| requirements=depset.requirements, | ||
| constraints=depset.constraints, | ||
| name=depset.name, | ||
| output=depset.output, | ||
| append_flags=depset.append_flags, | ||
| override_flags=depset.override_flags, | ||
| ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The relax method call is missing the include_setuptools argument. This means the include_setuptools setting from the depset definition will be ignored, and it will always default to False.
| self.relax( | |
| source_depset=depset.source_depset, | |
| drop_packages=depset.drop_packages, | |
| requirements=depset.requirements, | |
| constraints=depset.constraints, | |
| name=depset.name, | |
| output=depset.output, | |
| append_flags=depset.append_flags, | |
| override_flags=depset.override_flags, | |
| ) | |
| self.relax( | |
| source_depset=depset.source_depset, | |
| drop_packages=depset.drop_packages, | |
| requirements=depset.requirements, | |
| constraints=depset.constraints, | |
| name=depset.name, | |
| output=depset.output, | |
| append_flags=depset.append_flags, | |
| override_flags=depset.override_flags, | |
| include_setuptools=depset.include_setuptools, | |
| ) |
| temp_dir = tempfile.mkdtemp() | ||
| relaxed_output_file = Path(temp_dir) / f"{Path(depset.output).name}.relaxed" | ||
| _graph_to_lockfile(dependency_graph.graph, relaxed_output_file) | ||
| requirements.append(relaxed_output_file.as_posix()) | ||
| self.compile( | ||
| constraints=constraints, | ||
| requirements=requirements, | ||
| name=name, | ||
| output=output, | ||
| append_flags=append_flags, | ||
| override_flags=override_flags, | ||
| include_setuptools=include_setuptools, | ||
| ) | ||
| shutil.rmtree(temp_dir) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using with tempfile.TemporaryDirectory() ensures that the temporary directory is cleaned up even if an error occurs within the block. This is more robust than manually calling shutil.rmtree.
Additionally, it's better to avoid modifying the input requirements list directly. Creating a new list new_requirements is a safer approach that prevents unexpected side effects.
| temp_dir = tempfile.mkdtemp() | |
| relaxed_output_file = Path(temp_dir) / f"{Path(depset.output).name}.relaxed" | |
| _graph_to_lockfile(dependency_graph.graph, relaxed_output_file) | |
| requirements.append(relaxed_output_file.as_posix()) | |
| self.compile( | |
| constraints=constraints, | |
| requirements=requirements, | |
| name=name, | |
| output=output, | |
| append_flags=append_flags, | |
| override_flags=override_flags, | |
| include_setuptools=include_setuptools, | |
| ) | |
| shutil.rmtree(temp_dir) | |
| with tempfile.TemporaryDirectory() as temp_dir: | |
| relaxed_output_file = Path(temp_dir) / f"{Path(depset.output).name}.relaxed" | |
| _graph_to_lockfile(dependency_graph.graph, relaxed_output_file) | |
| new_requirements = requirements + [relaxed_output_file.as_posix()] | |
| self.compile( | |
| constraints=constraints, | |
| requirements=new_requirements, | |
| name=name, | |
| output=output, | |
| append_flags=append_flags, | |
| override_flags=override_flags, | |
| include_setuptools=include_setuptools, | |
| ) |
| ) | ||
| + "\n" | ||
| ) | ||
| print(f"Writing lockfile to {output_path}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| deps_arr = [] | ||
| name = None | ||
| version = None | ||
| for line in f.readlines(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| package_line_match = re.search( | ||
| r"([A-Za-z0-9_.-]+)==([A-Za-z0-9.+-]+)", line | ||
| ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using re.match with line.lstrip() is more robust for parsing package lines. re.search can find a match anywhere in the line, which might lead to incorrect parsing if a similar pattern appears in a comment or elsewhere. re.match ensures the pattern is at the beginning of the (stripped) line, which is expected for a package declaration.
| package_line_match = re.search( | |
| r"([A-Za-z0-9_.-]+)==([A-Za-z0-9.+-]+)", line | |
| ) | |
| package_line_match = re.match( | |
| r"([A-Za-z0-9_.-]+)==([A-Za-z0-9.+-]+)", line.lstrip() | |
| ) |
| output=depset.output, | ||
| append_flags=depset.append_flags, | ||
| override_flags=depset.override_flags, | ||
| ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing include_setuptools parameter in relax call
Medium Severity
The call to self.relax() in execute_depset doesn't pass include_setuptools=depset.include_setuptools, unlike all other operations (compile, subset, expand) which consistently pass this parameter. The relax method accepts and uses include_setuptools, but it will always receive the default value of False instead of the user-configured value. This means include_setuptools: true in a relax depset configuration will be silently ignored.
| temp_dir = tempfile.mkdtemp() | ||
| relaxed_output_file = Path(temp_dir) / f"{Path(depset.output).name}.relaxed" | ||
| _graph_to_lockfile(dependency_graph.graph, relaxed_output_file) | ||
| requirements.append(relaxed_output_file.as_posix()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Relax method mutates input requirements list
Low Severity
The relax method directly mutates the input requirements list with requirements.append(), which modifies depset.requirements from the caller. This is inconsistent with expand, which creates a new list before modifying it. After shutil.rmtree(temp_dir) is called, depset.requirements contains a path to a deleted temp file, which could cause issues if the depset is accessed later or in retry scenarios.
| ) | ||
| + "\n" | ||
| ) | ||
| print(f"Writing lockfile to {output_path}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Debug print statement instead of click.echo
Low Severity
The _graph_to_lockfile function uses print() to output "Writing lockfile to {output_path}" while the rest of the codebase consistently uses click.echo() for user output. This outputs a temporary file path that gets deleted immediately after by shutil.rmtree(), making this appear to be accidentally committed debug code rather than intentional user feedback.
| override_flags=override_flags, | ||
| include_setuptools=include_setuptools, | ||
| ) | ||
| shutil.rmtree(temp_dir) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Temp directory not cleaned up if compile fails
Low Severity
The relax method creates a temp directory with tempfile.mkdtemp() but doesn't use try/finally to ensure cleanup. If self.compile() raises a RuntimeError (which occurs when the uv command fails), shutil.rmtree(temp_dir) is never called and the temp directory is orphaned. This could accumulate orphaned directories on disk if relax operations fail repeatedly.
Adding relax operation and unit tests