Specify minio env vars in integration tests #3001
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
eapolinario
requested review from
wild-endeavor,
kumare3,
pingsutw,
cosmicBboy,
samhita-alla,
thomasjpfan and
Future-Outlier
as code owners
wild-endeavor
approved these changes
Dec 13, 2024
shuyingliang
pushed a commit
to shuyingliang/flytekit
that referenced
this pull request
Dec 14, 2024
Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Shuying Liang <shuying.liang@gmail.com>
wild-endeavor
pushed a commit
that referenced
this pull request
Dec 16, 2024
Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
3 tasks
JiangJiaWei1103
added a commit
to JiangJiaWei1103/flytekit
that referenced
this pull request
Jan 2, 2025
Please refer to flyteorg#3001 Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com>
Future-Outlier
pushed a commit
that referenced
this pull request
Jan 3, 2025
* fix: Open FlyteFile from remote path
Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com>
* Add integration test
Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com>
* refactor: Use ctx as param instead of recreation
Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com>
* refactor: Clean test logic
1. Remove redundant prints
2. Use `mock.patch.dict` to setup `os.environ` for the current test fn
* Avoid contaminating other tests running in the same process
Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com>
* refactor: Setup local path and downloader in constructor
Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com>
* refactor: Move SimpleFileTransfer to an utility file
Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com>
* Remove redundant env var setup
Please refer to #3001
Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com>
* test: Add another ff use case
Create ff in one task pod and read it in another task pod.
Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com>
---------
Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com>
3 tasks
granthamtaylor
added a commit
that referenced
this pull request
Jan 6, 2025
* Store protos in local cache (#3022) * Store proto obj instead of model Literal in local cache Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove unused file Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Bump aiohttp from 3.9.5 to 3.10.11 (#3018) Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.9.5 to 3.10.11. - [Release notes](https://github.com/aio-libs/aiohttp/releases) - [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst) - [Commits](aio-libs/aiohttp@v3.9.5...v3.10.11) --- updated-dependencies: - dependency-name: aiohttp dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix bug in FlyteDirectory.listdir on local files (#2926) * Fix issue in FlyteDirectory.listdir Fixes flyteorg/flyte#6005 Signed-off-by: Pim de Haan <pim@cusp.ai> * Added test Signed-off-by: Pim de Haan <pim@cusp.ai> * Run make lint Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> --------- Signed-off-by: Pim de Haan <pim@cusp.ai> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Fix unit tests in airflow plugin (#3024) Signed-off-by: Kevin Su <pingsutw@apache.org> * fix: Fix resource meta typos for async agent (#3023) Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * fix: format commands output (#3026) * Fix pydantic basemodel default input (#3013) * Fix pydantic default input Signed-off-by: Future-Outlier <eric901201@gmail.com> * add pydantic integration test Signed-off-by: Future-Outlier <eric901201@gmail.com> * Use duck typing by Thomas's advice Signed-off-by: Future-Outlier <eric901201@gmail.com> Co-authored-by: Thomas J. Fan <thomasjpfan@gmail.com> * lint Signed-off-by: Future-Outlier <eric901201@gmail.com> --------- Signed-off-by: Future-Outlier <eric901201@gmail.com> Co-authored-by: Thomas J. Fan <thomasjpfan@gmail.com> * [BUG] Open FlyteFile from remote path (#2991) * fix: Open FlyteFile from remote path Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * Add integration test Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * refactor: Use ctx as param instead of recreation Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * refactor: Clean test logic 1. Remove redundant prints 2. Use `mock.patch.dict` to setup `os.environ` for the current test fn * Avoid contaminating other tests running in the same process Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * refactor: Setup local path and downloader in constructor Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * refactor: Move SimpleFileTransfer to an utility file Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * Remove redundant env var setup Please refer to #3001 Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * test: Add another ff use case Create ff in one task pod and read it in another task pod. Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> --------- Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * vllm inference plugin (#2967) * vllm inference plugin Signed-off-by: Daniel Sola <daniel.sola@union.ai> * fixed default value Signed-off-by: Daniel Sola <daniel.sola@union.ai> --------- Signed-off-by: Daniel Sola <daniel.sola@union.ai> * Add poetry to image spec (#3025) * Add poetry to image spec Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> * Add stricter check Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> --------- Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> * [test] Add integration test for accessing sd sttr in dc (#2969) * test: Add integration test for attr access of sd Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * Correct file path Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * test: Support interaction with minio s3 bucket 1. Upload a local parquet file to minio s3 bucket 2. Access StructuredDataset attr from a dataclass 3. Open StructuredDataset from a remote path Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * Delete an unmerged integration test Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * Try imagespec with commit sha of corresponding fix Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * Remove redundant test Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * Remove default_factory and create sd dc from input uri Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * refactor: Clean test logic 1. Remove redundant prints 2. Use `mock.patch.dict` to setup `os.environ` for the current test fn * Avoid contaminating other tests running in the same process Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * Remove redundant minio env var setup and add test comments Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * Support uploading tmp pqt file Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * Udpate deprecated module Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> * Remove redundant and unused imports Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> --------- Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> --------- Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Pim de Haan <pim@cusp.ai> Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: JiaWei Jiang <waynechuang97@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> Signed-off-by: Daniel Sola <daniel.sola@union.ai> Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pim de Haan <pimdehaan@gmail.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Co-authored-by: 江家瑋 <36886416+JiangJiaWei1103@users.noreply.github.com> Co-authored-by: V <0426vincent@gmail.com> Co-authored-by: Han-Ru Chen (Future-Outlier) <eric901201@gmail.com> Co-authored-by: Thomas J. Fan <thomasjpfan@gmail.com> Co-authored-by: Daniel Sola <40698988+dansola@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
In flyteorg/flyte#5947 we stopped emitting the config the storage section to instruct clients to use the minio credentials. This was released in flytectl 0.9.3 (which was yanked due in flyteorg/flyte#6100) and present in 0.9.4.
What changes were proposed in this pull request?
We specify environment variables to instruct the flytekit boto client to hit minio.
How was this patch tested?
Ran tests locally.
Setup process
Screenshots
Check all the applicable boxes
Related PRs
Docs link