-
Notifications
You must be signed in to change notification settings - Fork 146
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for ns #1188
base: main
Are you sure you want to change the base?
Support for ns #1188
Conversation
@kevinjqliu can you please review my PR and let me know if i made a mistake (please check with extra caution as it is my first contribution to any open source project) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the contribution! Left a few comments
@@ -1068,8 +1068,17 @@ def primitive(self, primitive: pa.DataType) -> PrimitiveType: | |||
return StringType() | |||
elif pa.types.is_date32(primitive): | |||
return DateType() | |||
elif isinstance(primitive, pa.Time64Type) and primitive.unit == "us": | |||
elif isinstance(primitive, pa.Time64Type) and (primitive.unit == "us" or primitive.unit == "ns"): | |||
primitive = cast(pa.TimestampType, primitive) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Time64Type
and TimestampType
and different, dont need to cast here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@@ -1068,8 +1068,17 @@ def primitive(self, primitive: pa.DataType) -> PrimitiveType: | |||
return StringType() | |||
elif pa.types.is_date32(primitive): | |||
return DateType() | |||
elif isinstance(primitive, pa.Time64Type) and primitive.unit == "us": | |||
elif isinstance(primitive, pa.Time64Type) and (primitive.unit == "us" or primitive.unit == "ns"): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think we can get rid fo the us/ns check here, since Time64Type only supports those 2 https://arrow.apache.org/docs/python/generated/pyarrow.Time64Type.html
primitive = cast(pa.TimestampType, primitive) | ||
if primitive.unit == "ns": | ||
if self._downcast_ns_timestamp_to_us: | ||
logger.warning("Iceberg does not yet support 'ns' timestamp precision. Downcasting to 'us'.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lets add a test to verify that when _downcast_ns_timestamp_to_us
is True, the downcasting happens
added support for ns and enabled downcasting similar to the similar to the
pa.types.is_timestamp(primitive):
, didn't enable upcasting as it wasn't mentioned in the issue