Adding support of TIME64 datatype (attempt 2) #3697
Draft
+258
−95
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
❌ 10 Tests Failed:
View the top 3 failed test(s) by shortest run time
View the full list of 2 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
jgao54
force-pushed
the
TIME64
branch
11 times, most recently
from
f9e2eb2 to
3eb10eb
Compare
jgao54
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Postgres TIME supports range: [00:00:00, 24:00:00], with microsecond precision
MySQL TIME supports range: [-838:59:59.000000, 838:59:59.999999], with microsecond precision
ClickHouse Time64 supports range: [-999:59:59.000000, 999:59:59.999999], with nanosecond precision
In theory ClickHouse could fully support Postgres and MySQL timestamps as is. However, the current implementation converts MySQL time value is in the range of [00:00:00, 24:00:00].
One subtle, but IMO useful improvement from this PR (aside from adding Time64 support) that's worth calling out:
we used to have Postgres convert to QValueTime while MySQL convert to QValueTimestamp prior to Time64 was introduced; meaning Posgres handles conversion to DateTime64 in the destination connector, while MySQL handles the conversion in the source connector. With this PR, conversion is consistently handled in the destination connector. So we always convert Time -> QValueTime, and then handle backwards-compatibility by optionally converting QValueTime -> DateTime64 if on an older version of peerdb or if on an older version of ClickHouse that does not support Time64. This is possible because
toTime64andparseDateTime64BestEffortOrNullboth support parsing string with `hh:mm:ss'.I've also evaluated bringing consistency by aligning Postgres to MySQL, and do the branching when mapping Time (source type) -> QValue, and keep the destination consistent instead. This is a messier approach, because it means we need to know ClickHouse version when we are handling the branching (since we can't convert to Time64 unless it is supported by ClickHouse), so the source connector needs to know about ClickHouse version which is not a concept available today.
That said, if the business for supporting MySQL's handling is strong, the fix would be to implement a custom conversion from duration to int64 representing milliseconds (instead of hh:mm:ss string), and then have MySQL optionally handle that case based on the source colType. It's do-able, just more edge case handling involved; so I would prefer not unless absolutely necessary.