Get query logs from unity catalog

[usefulalgorithm]

🤔 Why?

• To get query logs and relevant information from databricks system.
• Use newer databricks SDK.
  (SC links: SC21951, SC21623)

🤓 What?

• updated to use new sdk.
• get query logs with WorkspaceClient.queriy_history.list, and return those logs along with the datasets.

🧪 Tested?

• Modified and ran unit tests.
• Tested with staging configurations. Was able to extract query logs with the new SDK. See description for sample query log entry.

Description

A sample query log entry:

        {
          "_id": "UNITY_CATALOG:01ee7724-1377-18f6-9781-b0aa2bb3fd13",
          "bytesRead": 0.0,
          "bytesWritten": 0.0,
          "duration": 33686.0,
          "platform": "UNITY_CATALOG",
          "queryId": "01ee7724-1377-18f6-9781-b0aa2bb3fd13",
          "rowsRead": 0.0,
          "rowsWritten": 3.0,
          "sql": "-- this is a system generated query from catalog explorer\ndescribe history `main`.`default`.`testing` LIMIT 100;",
          "sqlHash": "fd1d327232c9df5c354a6c7cd51276a7",
          "startTime": "2023-10-30T12:58:52.420000+00:00",
          "userId": "dev@metaphor.io"
        }

[shortcut-integration]

This pull request has been linked to Shortcut Story #21951: Get query log from Unity Catalog.

[usefulalgorithm]

TODO

• add config values in the crawler configs: something like

@dataclass(config=ConnectorConfig)
class SnowflakeQueryLogConfig:
    # Number of days back of query logs to fetch, if 0, don't fetch query logs
    lookback_days: int = 1

    # Query log filter to exclude certain usernames
    excluded_usernames: Set[str] = field(default_factory=lambda: set())

    # The number of query logs to fetch from Snowflake in one batch
    fetch_size: int = DEFAULT_QUERY_LOG_FETCH_SIZE

    # Queries larger than this size will not be processed
    max_query_size: int = DEFAULT_MAX_QUERY_SIZE

• Test on a real environment that new list_table/column_lineage works

[github-actions]

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
12470	11278	90%	85%	🟢

New Files

No new covered files...

Modified Files

File	Coverage	Status
metaphor/unity_catalog/config.py	100%	🟢
metaphor/unity_catalog/extractor.py	93%	🟢
metaphor/unity_catalog/models.py	100%	🟢
metaphor/unity_catalog/utils.py	100%	🟢
TOTAL	98%	🟢

updated for commit: 26a236a by action🐍

[codecov]

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Files	Coverage Δ
metaphor/unity_catalog/config.py	100.00% <100.00%> (ø)
metaphor/unity_catalog/models.py	100.00% <100.00%> (+1.31%)	⬆️
metaphor/unity_catalog/utils.py	100.00% <100.00%> (ø)
metaphor/unity_catalog/extractor.py	93.15% <92.75%> (+1.15%)	⬆️

... and 31 files with indirect coverage changes

📢 Thoughts on this report? Let us know!.

[mars-lan]

Please also update https://github.com/MetaphorData/connectors/blob/main/metaphor/unity_catalog/README.md with the new configs and the requirement of Databricks Premium or above.

[mars-lan]

Please also include a sample entry of the query log in the description.

[mars-lan]

LGTM at a high level. Will let @elic-eon give the final ship it.