Add poetry to image spec #3025

thomasjpfan · 2024-12-30T19:30:34Z

Tracking issue

Why are the changes needed?

This allows poetry uses to create a image spec with their dependencies. This PR treats poetry as a "dependency manager". This means that their local project will not be installed. If we see that poetry users want to install their local package, then we can extend this feature with a follow up PR.

What changes were proposed in this pull request?

This PR adds a requirements="poetry.lock" that creates an ImageSpec with poetry.

How was this patch tested?

I created a repo with poetry init with this workflow:

from flytekit import task, ImageSpec
import numpy as np
from random import choice
from string import ascii_letters


image = ImageSpec(
    name="numpy-poetry",
    requirements="poetry.lock",
    registry="localhost:30000",
    env={"A": "".join(choice(ascii_letters) for _ in range(20))},
)


@task(container_image=image)
def add_task() -> np.ndarray:
    x = np.ones(10)
    return x + 10

and ran pyflyte run --remote workflows/main.py.

Summary by Bito

This PR implements Poetry support in ImageSpec with lock file handling capabilities and Docker build integration. It enhances array node capabilities with ReferenceTask support and improves file handling for remote access. The implementation includes comprehensive error handling across components, particularly in file operations and logging systems. Changes include expanded testing infrastructure with improved variable naming and VSCode integration enhancements.

Unit tests added: True

Estimated effort to review (1-5, lower is better): 5

Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>

flyte-bot · 2024-12-30T19:30:49Z

Code Review Agent Run #73368b

Actionable Suggestions - 5

tests/flytekit/unit/core/image_spec/test_default_builder.py - 2
- Consider renaming shadowed lock_file variable · Line 281-281
- Consider consolidating duplicate pytest parametrize decorator · Line 275-275
flytekit/image_spec/default_builder.py - 3
- Consider single $ for PATH variable · Line 58-58
- Consider moving poetry arg to template · Line 196-197
- Consider validating pyproject.toml existence · Line 216-217

Review Details

Files reviewed - 2 · Commit Range: 0813ae8..0813ae8
- flytekit/image_spec/default_builder.py
- tests/flytekit/unit/core/image_spec/test_default_builder.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

flyte-bot · 2024-12-30T19:35:03Z

Changelist by Bito

This pull request implements the following key changes.

Key Change	Files Impacted
Feature Improvement - Poetry Support for ImageSpec	- `default_builder.py` - Added Poetry lock file support and template for Docker builds - `test_default_builder.py` - Added test cases for Poetry lock file handling and error scenarios

flyte-bot · 2024-12-30T19:35:05Z

tests/flytekit/unit/core/image_spec/test_default_builder.py

    run_mock = Mock()
    monkeypatch.setattr("flytekit.image_spec.default_builder.run", run_mock)

-    uv_lock_file = tmp_path / "uv.lock"
-    uv_lock_file.write_text("this is a lock file")
+    lock_file = tmp_path / lock_file


Consider renaming shadowed lock_file variable

The variable name lock_file is being redefined in the function scope, shadowing the parameter with the same name. This could lead to confusion and potential bugs.

Code suggestion

Check the AI-generated fix before applying

Suggested change

lock_file = tmp_path / lock_file

lock_file_path = tmp_path / lock_file

Code Review Run #73368b

Is this a valid issue, or was it incorrectly flagged by the Agent?

it was incorrectly flagged

flyte-bot · 2024-12-30T19:35:06Z

tests/flytekit/unit/core/image_spec/test_default_builder.py

        builder.build_image(image_spec)


+@pytest.mark.parametrize("lock_file", ["uv.lock", "poetry.lock"])


Consider consolidating duplicate pytest parametrize decorator

Consider consolidating the duplicate @pytest.mark.parametrize decorator with the one at line 254 since both test functions use the same parameters.

Code suggestion

Check the AI-generated fix before applying

@@ -254,7 +254,6 @@ @pytest.mark.parametrize("lock_file", ["uv.lock", "poetry.lock"]) @pytest.mark.filterwarnings("ignore::UserWarning") def test_lock_errors_no_pyproject_toml(monkeypatch, tmp_path, lock_file): @@ -274,7 +274,6 @@ -@pytest.mark.parametrize("lock_file", ["uv.lock", "poetry.lock"]) @pytest.mark.filterwarnings("ignore::UserWarning") def test_uv_lock_error_no_packages(monkeypatch, tmp_path, lock_file):

Code Review Run #73368b

Is this a valid issue, or was it incorrectly flagged by the Agent?

it was incorrectly flagged

flytekit/image_spec/default_builder.py

Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>

flyte-bot · 2024-12-30T20:41:17Z

Code Review Agent Run #edb466

Actionable Suggestions - 0

Review Details

Files reviewed - 1 · Commit Range: 0813ae8..b32a4df
- tests/flytekit/unit/core/image_spec/test_default_builder.py
Files skipped - 0
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>

flyte-bot · 2025-01-03T18:18:59Z

Code Review Agent Run #9afb0f

Actionable Suggestions - 0

Additional Suggestions - 7

flytekit/clis/sdk_in_container/run.py - 1
- Consider consolidating JSON serialization logic · Line 478-486
flytekit/core/array_node.py - 1
- Consider adding data mode validation check · Line 140-141
tests/flytekit/unit/cli/pyflyte/test_run.py - 2
- Consider checking exact message occurrences · Line 480-480
- Consider more precise output assertions · Line 853-854
tests/flytekit/integration/remote/test_remote.py - 2
- Consider using unittest.mock instead of mock · Line 17-17
- Consider adding cleanup error handling · Line 832-835
tests/flytekit/unit/bin/test_python_entrypoint.py - 1
- Consider consolidating repeated timestamp initialization pattern · Line 529-540

Review Details

Files reviewed - 28 · Commit Range: b32a4df..fa746c8
- dev-requirements.txt
- flytekit/clis/sdk_in_container/run.py
- flytekit/core/array_node.py
- flytekit/core/array_node_map_task.py
- flytekit/core/context_manager.py
- flytekit/core/local_cache.py
- flytekit/core/tracker.py
- flytekit/core/workflow.py
- flytekit/extend/backend/agent_service.py
- flytekit/extend/backend/base_agent.py
- flytekit/interactive/vscode_lib/decorator.py
- flytekit/loggers.py
- flytekit/models/core/workflow.py
- flytekit/models/literals.py
- flytekit/tools/translator.py
- flytekit/types/directory/types.py
- flytekit/types/file/file.py
- plugins/flytekit-airflow/setup.py
- plugins/flytekit-flyteinteractive/tests/test_flyteinteractive_vscode.py
- pyproject.toml
- tests/flytekit/integration/remote/test_remote.py
- tests/flytekit/integration/remote/utils.py
- tests/flytekit/integration/remote/workflows/basic/flytefile.py
- tests/flytekit/integration/remote/workflows/basic/pydantic_wf.py
- tests/flytekit/unit/bin/test_python_entrypoint.py
- tests/flytekit/unit/cli/pyflyte/test_run.py
- tests/flytekit/unit/core/test_local_cache.py
- tests/flytekit/unit/types/directory/test_listdir.py
Files skipped - 1
- .github/workflows/pythonpublish.yml - Reason: Filter setting
Tools
- Whispers (Secret Scanner) - ✔︎ Successful
- Detect-secrets (Secret Scanner) - ✔︎ Successful
- MyPy (Static Code Analysis) - ✔︎ Successful
- Astral Ruff (Static Code Analysis) - ✔︎ Successful

AI Code Review powered by

Add poetry to image spec

0813ae8

Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>

thomasjpfan requested review from wild-endeavor, kumare3, eapolinario, pingsutw, cosmicBboy, samhita-alla and Future-Outlier as code owners December 30, 2024 19:30

flyte-bot reviewed Dec 30, 2024

View reviewed changes

flytekit/image_spec/default_builder.py Show resolved Hide resolved

flyte-bot reviewed Dec 30, 2024

View reviewed changes

flytekit/image_spec/default_builder.py Show resolved Hide resolved

flyte-bot reviewed Dec 30, 2024

View reviewed changes

flytekit/image_spec/default_builder.py Show resolved Hide resolved

Add stricter check

b32a4df

Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>

wild-endeavor approved these changes Jan 2, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/master' into flytekit-2

fa746c8

Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add poetry to image spec #3025

Add poetry to image spec #3025

thomasjpfan commented Dec 30, 2024 •

edited by flyte-bot

Loading

flyte-bot commented Dec 30, 2024 •

edited

Loading

Code Review Agent Run #73368b

flyte-bot commented Dec 30, 2024 •

edited

Loading

Changelist by Bito

flyte-bot Dec 30, 2024

flyte-bot Dec 30, 2024

flyte-bot commented Dec 30, 2024 •

edited

Loading

Code Review Agent Run #edb466

flyte-bot commented Jan 3, 2025 •

edited

Loading

Code Review Agent Run #9afb0f

	lock_file = tmp_path / lock_file
	lock_file_path = tmp_path / lock_file

		builder.build_image(image_spec)


		@pytest.mark.parametrize("lock_file", ["uv.lock", "poetry.lock"])

Add poetry to image spec #3025

Are you sure you want to change the base?

Add poetry to image spec #3025

Conversation

thomasjpfan commented Dec 30, 2024 • edited by flyte-bot Loading

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Summary by Bito

flyte-bot commented Dec 30, 2024 • edited Loading

Code Review Agent Run #73368b

flyte-bot commented Dec 30, 2024 • edited Loading

Changelist by Bito

flyte-bot Dec 30, 2024

Choose a reason for hiding this comment

flyte-bot Dec 30, 2024

Choose a reason for hiding this comment

flyte-bot commented Dec 30, 2024 • edited Loading

Code Review Agent Run #edb466

flyte-bot commented Jan 3, 2025 • edited Loading

Code Review Agent Run #9afb0f

thomasjpfan commented Dec 30, 2024 •

edited by flyte-bot

Loading

flyte-bot commented Dec 30, 2024 •

edited

Loading

flyte-bot commented Dec 30, 2024 •

edited

Loading

flyte-bot commented Dec 30, 2024 •

edited

Loading

flyte-bot commented Jan 3, 2025 •

edited

Loading