Semantic model perf lab #426
Conversation
| return str(self._properties) | ||
|
|
||
|
|
||
| def _get_or_create_workspace( |
There was a problem hiding this comment.
should move to common utils.
There was a problem hiding this comment.
Following up offline with Michael.
There was a problem hiding this comment.
Moving this to _helper_functions.py
| create_abfss_path, | ||
| ) | ||
|
|
||
| class PropertyBag: |
There was a problem hiding this comment.
Yeah, dict is much easier. Replaced.
| raise ValueError("For new workspaces, the workspace parameter must be string, not a Guid. Please provide a workspace name.") | ||
| except ValueError: | ||
| # OK, it's not a Guid. But also make sure the workspace parameter isn't empty. | ||
| if workspace == "" or workspace is None: |
There was a problem hiding this comment.
just use if workspace:
https://stackoverflow.com/questions/9573244/how-to-check-if-the-string-is-empty-in-python
There was a problem hiding this comment.
Thank you. Changed throughout.
| print(f"{icons.green_dot} Workspace '{workspace_name}' created.") | ||
| return (workspace,workspace_id) | ||
|
|
||
| def _get_or_create_lakehouse( |
There was a problem hiding this comment.
Following up offline with Michael.
There was a problem hiding this comment.
Moving this to _helper_functions.py
| """ | ||
|
|
||
| # Treat empty strings as None. | ||
| if lakehouse == "": |
There was a problem hiding this comment.
not sure why this is needed.
There was a problem hiding this comment.
No longer in the sample_lab file..
|
|
||
| try: | ||
| (target_workspace_name, target_workspace_id) = resolve_workspace_name_and_id(workspace=target_workspace) | ||
| except: |
There was a problem hiding this comment.
catch specific exception, otherwise we mask errors
There was a problem hiding this comment.
Thanks. Replaced with (WorkspaceNotFoundException, ValueError).
|
|
||
| return spark.createDataFrame(rows, schema=schema).dropDuplicates() | ||
|
|
||
| def _get_min_max_keys( |
There was a problem hiding this comment.
Following up offline with Michael.
There was a problem hiding this comment.
Michael already has a _get_column_aggregate utility function which does essentially the same thing, but it doesn't take a table path as a parameter and only returns a single int value. For now, let's leave it here. Happy to move later if OK with Michael.
| def get_storage_table_column_segments( | ||
| test_cycle_definitions: DataFrame, | ||
| tables_info: DataFrame | ||
| )->DataFrame: |
There was a problem hiding this comment.
run black to get formatting
|
|
||
| try: | ||
| (target_workspace_name, target_workspace_id) = resolve_workspace_name_and_id(workspace=target_workspace) | ||
| except: |
There was a problem hiding this comment.
Fixed (WorkspaceNotFoundException, ValueError)
| ] | ||
|
|
||
|
|
||
| def _get_test_definitions( |
There was a problem hiding this comment.
the dataframe seems small, so I'd suggest not to use a pandas dataframe.
since you want to subsequently persist it (and maybe re-hydrate), I think a separate class would make it more convenient.
There was a problem hiding this comment.
Agreed. Classes are more user friendly and less error prone. Also let me hard-code the fields that the test definitions must have. But I do want to keep this flexible so that users can add more fields as needed for their specific cases. For example, a query Category can be useful for analysis but it's not strictly needed for the perf lab. Long story short, added TestDefinition and TestSuite classes that feature the standard fields but also persist and load any additional fields the user may want to have.
| olEPs = response.json().get("oneLakeEndpoints") | ||
| dfsEP = olEPs.get("dfsEndpoint") | ||
|
|
||
| start_expr = "let\n\tdatabase = " |
There was a problem hiding this comment.
multiline string using """
""" to improve readability
|
Closing this big PR for a series of small PRs, as discussed offline. |
Functions to provision and run a Semantic Model Perf Lab for continuous testing and ad-hoc investigations.