-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
⬆️ Upgrade libraries (pydantic v2) #6366
Merged
giancarloromeo
merged 210 commits into
ITISFoundation:pydantic_v2_migration
from
giancarloromeo:is4481/upgrade-libs
Oct 9, 2024
Merged
Changes from all commits
Commits
Show all changes
210 commits
Select commit
Hold shift + click to select a range
dcb5895
upgrade aws-library
giancarloromeo 3620668
upgrade dask-task-models-library
giancarloromeo 85dffd8
upgrade notifications-library
giancarloromeo 17b9457
upgrade postgres-database
giancarloromeo 9571d2f
upgrade service-library
giancarloromeo bff67e6
upgrade simcore-sdk
giancarloromeo 17681a5
upgrade settings-library
giancarloromeo 81c045b
upgrade service-integration
giancarloromeo 3f4901e
upgrade settings-library
giancarloromeo a0f2890
add type hint
giancarloromeo e1ac2c6
continue upgrading
giancarloromeo 92175f8
Merge branch 'pydantic_v2_migration' into is4481/upgrade-libs
giancarloromeo 7f3801a
continue upgrading
giancarloromeo a470e9e
continue upgrading
giancarloromeo 8652f39
update base requirements
giancarloromeo 7b3c39e
continue upgrading
giancarloromeo 2fe2b81
replace deprecated parse_obj methods
giancarloromeo f320883
replace deprecated dict methods
giancarloromeo 2b6180c
replace deprecated json methods
giancarloromeo c88b393
replace deprecated parse_obj_as methods
giancarloromeo ceb9f69
update model config access
giancarloromeo e0bc99b
update pydantic settings test
giancarloromeo 18b7e37
upgrade base module
giancarloromeo dc3d602
fix ByteSize values
giancarloromeo 724bed4
replace deprecated parse_obj_as method
giancarloromeo f2f007e
fix regex validation with look-ahead assertion
giancarloromeo 76f1d5e
fix validate tag_key
giancarloromeo 2112122
remove deprecated
giancarloromeo 7404f3e
continue upgrading
giancarloromeo 7e156d7
fix regex
giancarloromeo 1401f2e
fix error base class
giancarloromeo 5f553ce
continue upgrading
giancarloromeo 6ed0551
upgrade dependencies
giancarloromeo c36b542
fix error base class
giancarloromeo 49a0ab4
upgrade dependencies
giancarloromeo 0191717
continue upgrading
giancarloromeo 2e74551
upgrade dependencies
giancarloromeo c4e6fb2
fix email model validator
giancarloromeo 0846266
ignore code type
giancarloromeo 5601125
fix validate_call
giancarloromeo d47edd0
fix validate call
giancarloromeo 6f38de2
fix faker types
giancarloromeo e69b9ef
continue upgrading
giancarloromeo 9f17d51
continue upgrading
giancarloromeo d13f0e0
continue upgrading
giancarloromeo ef0e355
fix endpoint_url type
giancarloromeo 6c30f36
fix mock
giancarloromeo 3d734e0
fix endpoint_url type
giancarloromeo c667921
remove unused type ignore
giancarloromeo 4fb2bf2
update error base class
giancarloromeo c11f109
fix folders error
giancarloromeo 68fa4ae
use common osparcerrormixin
giancarloromeo 40a4627
continue upgrading
giancarloromeo 0563d81
upgrade dependencies
giancarloromeo 2094d4f
continue upgrading
giancarloromeo 57b0710
continue upgrading
giancarloromeo 29306b1
continue upgrading
giancarloromeo a9c3ef3
remove deprecated parse_obj_as
giancarloromeo 8250aff
fix url type
giancarloromeo 6c63ccf
fix url type
giancarloromeo fd2f6cb
fix rabbit settings
giancarloromeo 7a888f5
upgrade deprecated method
giancarloromeo b43955c
fix dns* creation
giancarloromeo a1a7e24
return redis dns as str
giancarloromeo d74f380
upgrade pydantic dependencies
giancarloromeo 40e34e5
continue upgrading
giancarloromeo 2843a12
continue upgrading
giancarloromeo c82afa5
fix anyhttpurl validation
giancarloromeo c96cd21
continue upgrading
giancarloromeo dfe63b5
continue upgrading
giancarloromeo 0df8f60
add ignore
giancarloromeo 56c1c89
fix errors messages
giancarloromeo 3b15198
continue upgrading
giancarloromeo bb10a3d
continue upgrading
giancarloromeo ec0404a
continue upgrading
giancarloromeo 96a0663
continue upgrading
giancarloromeo ad7ddb5
continue upgrading
giancarloromeo 6369b33
fix test
giancarloromeo 7d5537d
continue upgrading
giancarloromeo 486437d
update error msg
giancarloromeo 0167480
fix model_fields iteration
giancarloromeo 1409f87
fix labels type
giancarloromeo 94749e6
fix root field reference
giancarloromeo 223e2ac
continue upgrading
giancarloromeo a9d30ab
fix test
giancarloromeo 2a497f3
remove deprecated method
giancarloromeo 32318b0
fix error output
giancarloromeo 45baff8
fix base settings
giancarloromeo 81f80de
fix error
giancarloromeo 1c33bfb
fix error class
giancarloromeo 898155d
fix nullable
giancarloromeo e3fc9c5
update base
giancarloromeo e645331
update test
giancarloromeo ff5533b
restore Error type
giancarloromeo 0eaa8c8
update test
giancarloromeo 9c22d2b
force mode_rebuild
giancarloromeo 4e74ec7
fix port type
giancarloromeo 0e3f7d3
continue upgrading
giancarloromeo 6aaaea1
continue upgrading
giancarloromeo 4ddddc9
fix AnyUrl build
giancarloromeo 8120fff
fix field_info
giancarloromeo a28234d
fix utils service
giancarloromeo da97728
fix mypy
giancarloromeo 06912be
set optional
giancarloromeo a0d6f84
set optional
giancarloromeo 8e62f69
continue upgrading
giancarloromeo e3510cd
continue upgrading
giancarloromeo 3e8ca5d
continue upgrading
giancarloromeo fffb812
fix cached property
giancarloromeo e5f9263
remove pytest.mark
giancarloromeo bc208a2
fix mypy
giancarloromeo 85eb86f
fix parse none env
giancarloromeo 476e538
fix env list parsing
giancarloromeo f28c414
use BeforeValidator for country codes
giancarloromeo 5d4a6a2
fix mypy
giancarloromeo 1283488
fix utils cli
giancarloromeo 5cecc63
fix encoder issues
giancarloromeo cbc556f
add requirements
giancarloromeo 4c27843
skip missing data
giancarloromeo 9c4e07e
fix import
giancarloromeo 6c58ae5
fix option name
giancarloromeo 76b5cba
continue ugrading
giancarloromeo 7021117
firx param name
giancarloromeo cd1fbc1
fix mypy
giancarloromeo 4a29d2e
fix mypy
giancarloromeo 532206e
fix rootmodel
giancarloromeo 2cddbd8
continue upgrading
giancarloromeo 32a5453
continue upgrading
giancarloromeo 57f65d5
fix serializer
giancarloromeo 02d950e
fix mypy
giancarloromeo 16a9740
run bump-pydantic
giancarloromeo 04509d3
fix int test
giancarloromeo 10a80b3
fix rabbit method name
giancarloromeo 3f91e30
remove deprecated
giancarloromeo 9ebf506
replace deprecated dict
giancarloromeo 94c33aa
fix field
giancarloromeo 12ec0a3
upgrade requirements
giancarloromeo ca36539
fix pylint
giancarloromeo 4e49d2d
fix endopoints type
giancarloromeo f0fc1d7
fix pagination urls type
giancarloromeo aa6a73f
fix sonar
giancarloromeo 5debf98
fix code smells
giancarloromeo 9d93d7e
continue upgrading
giancarloromeo 0a1a1a4
continue upgrading
giancarloromeo 32622e0
fix mypy
giancarloromeo 9a6df80
move adapter out
giancarloromeo 8e53f7a
move adapter out
giancarloromeo 9f1c18c
simplify regex
giancarloromeo 1df1179
fix field name
giancarloromeo 9e99c51
fix validationerror match
giancarloromeo b07d12b
fix optional field
giancarloromeo 444a600
fix comments
giancarloromeo 2a8d6eb
remove redundant comment
giancarloromeo 386c316
remove unnecessary schema patch
giancarloromeo 68b1599
upgrade CI requrements
giancarloromeo 3aa9009
remove unnecessary cast
giancarloromeo 24679c1
add final
giancarloromeo c9f094f
fix ValidationError import
giancarloromeo fee36c0
fix mypy
giancarloromeo 79a4641
fix missing callback
giancarloromeo 06c8046
fix field name
giancarloromeo 323dbc9
code not used by base class
giancarloromeo 9a8e718
revert latest commit
giancarloromeo 42f07ea
set final
giancarloromeo 968b5ab
Add common library
giancarloromeo 67304b6
add code field
giancarloromeo 0ce5ef4
fix pylint
giancarloromeo 3f5d881
add base common library
giancarloromeo df864b4
add error classes
giancarloromeo 09c0073
point to common library
giancarloromeo d752d18
fix names
giancarloromeo 3f0ec86
remove moved modules
giancarloromeo 4a30d09
fix rename
giancarloromeo 1e57220
add utils
giancarloromeo cb7ac13
add common modules
giancarloromeo 30e2613
Merge branch 'add-common-library' into is4481/upgrade-libs
giancarloromeo 2663ed1
update requirements
giancarloromeo e358cd2
upgrade package name
giancarloromeo 0b8a6ab
remove moved modules
giancarloromeo 2b4e79d
update requirements
giancarloromeo e6efb6d
update requirements
giancarloromeo 51c6ca5
update requirements
giancarloromeo 4265fec
update package name
giancarloromeo 5a998a0
add requirement
giancarloromeo fc91024
add requirement
giancarloromeo 0ead0b4
update requirements
giancarloromeo 727c634
add py.typed
giancarloromeo f387ebb
add me as codeowner
giancarloromeo 5ed512a
update requirements
giancarloromeo 7bc8715
update requirements
giancarloromeo 18ad6d1
update requirements
giancarloromeo f63a537
minor fixes
giancarloromeo 97854a0
Merge branch 'add-common-library' into is4481/upgrade-libs
giancarloromeo 4b93c00
use anyhttpurl legacy
giancarloromeo 9114d67
Merge branch 'pydantic_v2_migration' into is4481/upgrade-libs
giancarloromeo 989e76a
upgrade reqs
giancarloromeo fa7072d
update reqs
giancarloromeo e36b18c
move changes
giancarloromeo fdd19c6
move base type adapters
giancarloromeo c913551
fix test
giancarloromeo 890583b
fix type
giancarloromeo ffd0adc
fix type
giancarloromeo 7663205
fix networks types
giancarloromeo 74330ed
update reqs
giancarloromeo 26348d0
fix type
giancarloromeo 6eb3bf7
fix type
giancarloromeo b3d0076
fix url types
giancarloromeo 4606e1f
add type hints
giancarloromeo a37efc1
fix minor
giancarloromeo 638a0be
add validation
giancarloromeo File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,18 +2,19 @@ | |
import re | ||
import tempfile | ||
from dataclasses import dataclass | ||
from typing import Any, ClassVar, TypeAlias | ||
from typing import Annotated, Final, TypeAlias | ||
|
||
import sh # type: ignore[import-untyped] | ||
from models_library.docker import DockerGenericTag | ||
from pydantic import ( | ||
BaseModel, | ||
ByteSize, | ||
ConstrainedStr, | ||
ConfigDict, | ||
Field, | ||
NonNegativeFloat, | ||
NonNegativeInt, | ||
validator, | ||
StringConstraints, | ||
field_validator, | ||
) | ||
from types_aiobotocore_ec2.literals import InstanceStateNameType, InstanceTypeType | ||
|
||
|
@@ -33,26 +34,26 @@ def __gt__(self, other: "Resources") -> bool: | |
return self.cpus > other.cpus or self.ram > other.ram | ||
|
||
def __add__(self, other: "Resources") -> "Resources": | ||
return Resources.construct( | ||
return Resources.model_construct( | ||
**{ | ||
key: a + b | ||
for (key, a), b in zip( | ||
self.dict().items(), other.dict().values(), strict=True | ||
self.model_dump().items(), other.model_dump().values(), strict=True | ||
) | ||
} | ||
) | ||
|
||
def __sub__(self, other: "Resources") -> "Resources": | ||
return Resources.construct( | ||
return Resources.model_construct( | ||
**{ | ||
key: a - b | ||
for (key, a), b in zip( | ||
self.dict().items(), other.dict().values(), strict=True | ||
self.model_dump().items(), other.model_dump().values(), strict=True | ||
) | ||
} | ||
) | ||
|
||
@validator("cpus", pre=True) | ||
@field_validator("cpus", mode="before") | ||
@classmethod | ||
def _floor_cpus_to_0(cls, v: float) -> float: | ||
return max(v, 0) | ||
|
@@ -67,19 +68,31 @@ class EC2InstanceType: | |
InstancePrivateDNSName: TypeAlias = str | ||
|
||
|
||
class AWSTagKey(ConstrainedStr): | ||
AWS_TAG_KEY_MIN_LENGTH: Final[int] = 1 | ||
AWS_TAG_KEY_MAX_LENGTH: Final[int] = 128 | ||
AWSTagKey: TypeAlias = Annotated[ | ||
# see [https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictions] | ||
regex = re.compile(r"^(?!(_index|\.{1,2})$)[a-zA-Z0-9\+\-=\._:@]+$") | ||
min_length = 1 | ||
max_length = 128 | ||
|
||
|
||
class AWSTagValue(ConstrainedStr): | ||
str, | ||
StringConstraints( | ||
min_length=AWS_TAG_KEY_MIN_LENGTH, | ||
max_length=AWS_TAG_KEY_MAX_LENGTH, | ||
pattern=re.compile(r"^(?!(_index|\.{1,2})$)[a-zA-Z0-9\+\-=\._:@]+$"), | ||
), | ||
] | ||
|
||
|
||
AWS_TAG_VALUE_MIN_LENGTH: Final[int] = 0 | ||
AWS_TAG_VALUE_MAX_LENGTH: Final[int] = 256 | ||
Comment on lines
+84
to
+85
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. declare on the top of the file |
||
AWSTagValue: TypeAlias = Annotated[ | ||
# see [https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictions] | ||
# quotes []{} were added as it allows to json encode. it seems to be accepted as a value | ||
regex = re.compile(r"^[a-zA-Z0-9\s\+\-=\.,_:/@\"\'\[\]\{\}]*$") | ||
min_length = 0 | ||
max_length = 256 | ||
str, | ||
StringConstraints( | ||
min_length=0, | ||
max_length=256, | ||
Comment on lines
+91
to
+92
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. maybe use these constants as well here |
||
pattern=r"^[a-zA-Z0-9\s\+\-=\.,_:/@\"\'\[\]\{\}]*$", | ||
), | ||
] | ||
|
||
|
||
EC2Tags: TypeAlias = dict[AWSTagKey, AWSTagValue] | ||
|
@@ -148,8 +161,23 @@ class EC2InstanceBootSpecific(BaseModel): | |
default=0, description="number of buffer EC2s to keep (defaults to 0)" | ||
) | ||
|
||
class Config: | ||
schema_extra: ClassVar[dict[str, Any]] = { | ||
@field_validator("custom_boot_scripts") | ||
@classmethod | ||
def validate_bash_calls(cls, v): | ||
try: | ||
with tempfile.NamedTemporaryFile(mode="wt", delete=True) as temp_file: | ||
temp_file.writelines(v) | ||
temp_file.flush() | ||
# NOTE: this will not capture runtime errors, but at least some syntax errors such as invalid quotes | ||
sh.bash("-n", temp_file.name) | ||
except sh.ErrorReturnCode as exc: | ||
msg = f"Invalid bash call in custom_boot_scripts: {v}, Error: {exc.stderr}" | ||
raise ValueError(msg) from exc | ||
|
||
return v | ||
|
||
model_config = ConfigDict( | ||
json_schema_extra={ | ||
"examples": [ | ||
{ | ||
# just AMI | ||
|
@@ -205,18 +233,4 @@ class Config: | |
}, | ||
] | ||
} | ||
|
||
@validator("custom_boot_scripts") | ||
@classmethod | ||
def validate_bash_calls(cls, v): | ||
try: | ||
with tempfile.NamedTemporaryFile(mode="wt", delete=True) as temp_file: | ||
temp_file.writelines(v) | ||
temp_file.flush() | ||
# NOTE: this will not capture runtime errors, but at least some syntax errors such as invalid quotes | ||
sh.bash("-n", temp_file.name) | ||
except sh.ErrorReturnCode as exc: | ||
msg = f"Invalid bash call in custom_boot_scripts: {v}, Error: {exc.stderr}" | ||
raise ValueError(msg) from exc | ||
|
||
return v | ||
) |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would dlclare them at the top of the file as it is common convention in python
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, forgot there :)