-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: Create workflow module scoped sagemaker_session to resolve test race condition #4518
Conversation
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #4518 +/- ##
==========================================
- Coverage 87.35% 87.35% -0.01%
==========================================
Files 389 389
Lines 36639 36639
==========================================
- Hits 32007 32006 -1
- Misses 4632 4633 +1 ☔ View full report in Codecov by Sentry. |
cd1bd9a
to
1dd00d9
Compare
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
Issue #, if available:
Workflow integ tests (e.g.
test_model_registration_with_model_repack
) have race condition with model_builder tests. The latter changes thesagemaker_session.settings._local_download_dir
when doing themodel_builder.build
and when the test is done, the tmp file in_local_download_dir
is cleaned up but the field is not reset, which blocks the workflow integ tests to upload model inference entry point file to S3. See the error thrown inhttps://github.com/aws/sagemaker-python-sdk/blame/c8d1428b1176e020c5225161d0426d3dc8853289/src/sagemaker/fw_utils.py#L453-L464
Description of changes: Create a separate sagemaker_session in workflow scope
Testing done: Integ tests
Merge Checklist
Put an
x
in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.General
Tests
unique_name_from_base
to create resource names in integ tests (if appropriate)By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.