source-snowflake: discover vector columns #2157
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Snowflake has a special type of column for storing vector embeddings. These columns contain either integer or float values in a fixed-length array. We can discover schemas for these fields accordingly, giving their discovered arrays annotations for the type of items and min/max length of the array, which will always be the size of the vector. These types of schematized fields may eventually be used by materializations for materializing into corresponding vector columns or indexes.
|
Setting this back to draft status since there is some uncertainty around how captures should be adding information like this to schemas, and there's not an immediate need for it. I'll keep the branch around just in case somebody shows up and wants to use it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
Snowflake has a special type of column for storing vector embeddings. These columns contain either integer or float values in a fixed-length array.
We can discover schemas for these fields accordingly, giving their discovered arrays annotations for the type of items and min/max length of the array, which will always be the size of the vector.
These types of schematized fields may eventually be used by materializations for materializing into corresponding vector columns or indexes.
Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)
Notes for reviewers:
(anything that might help someone review this PR)
This change is