chore: rename time zone tests #1830
Conversation
FBruzzesi
added
enhancement
| if "pyspark" in str(constructor): | ||
| data_ = { | ||
| col_name: [v.replace(tzinfo=timezone.utc) for v in col_values] | ||
| for col_name, col_values in data.items() | ||
| } |
There was a problem hiding this comment.
This is most likely needed for everyone not based in utc time. Since in pyspark_lazy_constructor we set .config("spark.sql.session.timeZone", "UTC"), the results will not be the same otherwise.
Context: it took me a bit to realize why instead of hours 12 and 2 I was getting 11 and 1
There was a problem hiding this comment.
shall we just put the tzinfo to timezone.utc when we define data=?
There was a problem hiding this comment.
I tried, and all pandas like backends raise warnings/errors sadly
| dates = {"a": [datetime(2001, 1, 1), None, datetime(2001, 1, 3)]} | ||
| dates = {"a": [datetime(2001, 1, 1), datetime(2001, 1, 3)]} |
There was a problem hiding this comment.
Removal of None is due to the fact that otherwise pyspark ends up creating a column of struct type. We might need to address this in the constructor?
There was a problem hiding this comment.
🤔 yeah might be good to see if we can address the constructor so it doesn't go via pandas, that might in general be good going forwards
| if "pyspark" in str(constructor): | ||
| data_ = { | ||
| col_name: [v.replace(tzinfo=timezone.utc) for v in col_values] | ||
| for col_name, col_values in data.items() | ||
| } |
There was a problem hiding this comment.
shall we just put the tzinfo to timezone.utc when we define data=?
| dates = {"a": [datetime(2001, 1, 1), None, datetime(2001, 1, 3)]} | ||
| dates = {"a": [datetime(2001, 1, 1), datetime(2001, 1, 3)]} |
There was a problem hiding this comment.
🤔 yeah might be good to see if we can address the constructor so it doesn't go via pandas, that might in general be good going forwards
MarcoGorelli
changed the title
What type of PR is this? (check all applicable)
Checklist