Add support for use_logical_type in write_pandas. #1706
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
use_logical_type is a new file format option of Snowflake. It is a Boolean that specifies whether Snowflake interprets Parquet logical types during data loading. The default behavior of write_pandas is unchanged. When users write a dataframe that contains datetimes with timezones and do not pass use_logical_type = True as an argument, a warning is raised (see #1687). Providing this option also fixes issue #1687
Please answer these questions before submitting your pull requests. Thanks!
What GitHub issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes SNOW-889573: write_pandas incorrectly writes timestamp_tz #1687
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
Snowflake recently released a fix for issue #1688, which was server side related. The fix involves a new file format parameter "use_logical_type". When writing data that contains timezones, the parameter should be set to True. The pull request presented here not only adds this option to the write_pandas method, but also raises a user warning if the users writes a pandas dataframe that contains timezones, but does not set use_logical_type to True. Reason for raising a warning and not an error is to maintain old behaviour, and to prevent users from suddenly running into issues when updating their pandas version.