Feature/remove session store #2219

Huongg · 2024-12-02T21:35:44Z

Description

Fixes #2208
This PR is part of the effort to remove experiment tracking by removing the session store from the back-end.

Main Changes:

Completely removed database.py.
Completely removed sqlite_store.py.
Removed session_store references from:
- data_loader.py
- server.py
Deleted all tests related to the session store.

QA Notes:
Please pull the changes locally and check that the app runs without errors,

Checklist

Read the contributing guidelines
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added new entries to the RELEASE.md file
Added tests to cover my changes

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

ravi-kumar-pilla · 2024-12-03T15:25:44Z

demo-project/src/demo_project/settings.py

-# SESSION_STORE_ARGS = {"path": str(Path(__file__).parents[2] / "data"),
-#                       "remote_path": "s3://{path-to-session_store}" }
-
-# Define custom context class. Defaults to `KedroContext`


I think you can keep the code from line 18-23 as is, as this is not related to ET

ravi-kumar-pilla · 2024-12-03T15:28:46Z

package/kedro_viz/database.py

@@ -21,21 +21,3 @@ def configure_wal_for_azure(engine):
    if is_azure_ml:
        with engine.connect() as conn:
            conn.execute(text("PRAGMA journal_mode=WAL;"))
-


This file is for handling SQLite which is not needed anymore, so I guess we can delete the file entirely

ravi-kumar-pilla · 2024-12-03T15:32:15Z

CONTRIBUTING.md

-```python
-from kedro_viz.integrations.kedro.sqlite_store import SQLiteStore
-SESSION_STORE_ARGS = {"path": str(Path(__file__).parents[2] / "data")}
-```


We need to make changes to all the starter templates (may be a note saying kedro-viz v11 does not support ET) as a follow up - https://github.com/kedro-org/kedro-starters/blob/main/spaceflights-pandas-viz/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/%7B%7B%20cookiecutter.python_package%20%7D%7D/settings.py

hey, @merelcht has already raised this concern, and I’ll create a ticket for it. For now, we’ve added a deprecation warning for Kedro datasets to alert users about this change coming in the next release.

ravi-kumar-pilla · 2024-12-03T15:37:22Z

Hi @Huongg ,

I know this PR is related to removing only Session store. I had a brief look and it looks fine for me. In general, I prepared a checklist as reference (you may add to this, but overall this would suffice) -

Remove graphql folder from api
Remove graphql_router from apps.py (line 66)
Remove “/et” from apps.py (line 84)
Remove SQLite Store from integrations/kedro/sqlite_store (This might also require changes to kedro starters as we say in the docs for ET, configure SQLiteStore in settings.py)
Remove docs related to ET

I hope this helps. I will also have a look at this PR once it is ready for review. Thank you

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

Huongg · 2024-12-04T13:54:04Z

Thanks, @ravi-kumar-pilla, for the helpful guidance! I’ve implemented your suggestion to update the code related to removing session_store. The changes related to GraphQL and the router will be addressed in a separate PR to keep this one more focused and easier to review. Please take a look and let me know your thoughts.

Currently, the test coverage isn’t at 100% because I’ve deleted all the graphql_tests. Since the GraphQL-related code will be removed soon, I’m wondering if it's okay to ignore this in the PR for now. Here's the link to the error for reference: https://github.com/kedro-org/kedro-viz/actions/runs/12160128272/job/33911739972?pr=2219

tynandebold · 2024-12-05T10:39:56Z

Is the plan to remove all of the graphql code, or only that which relates to experiment tracking?

I'm in favor of removing it entirely. There's one graphql request that's used on the flowchart page though, and that's the getVersion one. That could definitely be moved to a REST endpoint. Bringing up here so we don't forget about it.

Huongg · 2024-12-05T10:52:53Z

Is the plan to remove all of the graphql code, or only that which relates to experiment tracking?

I'm in favor of removing it entirely. There's one graphql request that's used on the flowchart page though, and that's the getVersion one. That could definitely be moved to a REST endpoint. Bringing up here so we don't forget about it.

hey so I'm thinking we're removing the graphql code related to ET in this PR, and I've opened a new issue separately to update the getVersion here

jitu5

Thanks @Huongg

SajidAlamQB

Awesome work @Huongg, I've looked through most of the changes and it LGTM!

…/remove-session-store Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

* [Frontend] Remove experiment tracking (#2212) * Remove exp tracking from feature hits and global toolbar; remove ExperimentsIcon Signed-off-by: tynandebold <thdebold@gmail.com> * Remove /experiment-tracking directory; fix tests Signed-off-by: tynandebold <thdebold@gmail.com> * Remove exp. tracking utils, reducers, styles Signed-off-by: tynandebold <thdebold@gmail.com> * Update moto requirement from ~=5.0.9 to ~=5.0.21 in /package (#2218) Updates the requirements on [moto](https://github.com/getmoto/moto) to permit the latest version. - [Release notes](https://github.com/getmoto/moto/releases) - [Changelog](https://github.com/getmoto/moto/blob/master/CHANGELOG.md) - [Commits](getmoto/moto@5.0.9...5.0.21) --- updated-dependencies: - dependency-name: moto dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update httpx requirement from ~=0.27.0 to ~=0.28.0 in /package (#2216) Updates the requirements on [httpx](https://github.com/encode/httpx) to permit the latest version. - [Release notes](https://github.com/encode/httpx/releases) - [Changelog](https://github.com/encode/httpx/blob/master/CHANGELOG.md) - [Commits](encode/httpx@0.27.0...0.28.0) --- updated-dependencies: - dependency-name: httpx dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update bandit requirement from ~=1.7 to ~=1.8 in /package (#2215) Updates the requirements on [bandit](https://github.com/PyCQA/bandit) to permit the latest version. - [Release notes](https://github.com/PyCQA/bandit/releases) - [Commits](PyCQA/bandit@1.7.0...1.8.0) --- updated-dependencies: - dependency-name: bandit dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update boto3 requirement from ~=1.34 to ~=1.35 in /package (#2214) Updates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version. - [Release notes](https://github.com/boto/boto3/releases) - [Commits](boto/boto3@1.34.0...1.35.71) --- updated-dependencies: - dependency-name: boto3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jitendra Gundaniya <38945204+jitu5@users.noreply.github.com> * Revert change from PR #2179 Signed-off-by: tynandebold <thdebold@gmail.com> * Remove the sanitizedPathname() function Signed-off-by: tynandebold <thdebold@gmail.com> * Remove Cypress tests for exp tracking Signed-off-by: tynandebold <thdebold@gmail.com> * Merge localStorageState state instead of the older allLocalStorageState Signed-off-by: tynandebold <thdebold@gmail.com> --------- Signed-off-by: tynandebold <thdebold@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jitendra Gundaniya <38945204+jitu5@users.noreply.github.com> * Feature/remove session store (#2219) * remove session store Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove relateds tests Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove doc about session store Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * lint formatting Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove session store from tests Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * update docs reference Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * update test_server Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove sessions store from load_data Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove make_db_session_factory Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove session store from _load_data_helper Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove database.py entirely Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove test graphql folder Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove cy test related to ET tab Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * Revert changes for .md as it will be done separately Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * revert contributing.md Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove e2e test failed from ET Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> --------- Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> Co-authored-by: Huong Nguyen <huong.nguyen@mckinsey.com> * Feature/remove-grahpql (#2222) * revert back the getVersion in graphql Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * update queries and schema Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * fix lint error Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * revert graphql_router since it's still using in getVersion Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove experiment-tracking.py, and runs and tracking datatset from repos Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * fix lint Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * update format lint Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove set_database_session Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> --------- Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> Co-authored-by: Huong Nguyen <huong.nguyen@mckinsey.com> * Feature/remove et doc (#2232) * revert back the getVersion in graphql Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * update queries and schema Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * fix lint error Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * revert graphql_router since it's still using in getVersion Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove experiment-tracking.py, and runs and tracking datatset from repos Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * fix lint Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * update format lint Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove set_database_session Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove ET mentioned in the doc and readme Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove ET and session store from readme Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * update new architecture diagram to remove apollo graphQL Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> --------- Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> Co-authored-by: Huong Nguyen <huong.nguyen@mckinsey.com> * Get version using restAPI (#2242) * set up rest for getversion Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * include get_latest_version, is_running_outdated_version Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * fixing the error Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove graphql Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * include test for getting version Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove check schema in lint Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * fix lint format error Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove test for graphql Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * fix syntax error Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove apollo Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * update libasound2 to libasound2t64 with the lates ubutun Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * update libgconf-2-4 Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * update versions to version Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> --------- Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> Co-authored-by: Huong Nguyen <huong.nguyen@mckinsey.com> * remove deprecation banner for ET Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * update release note Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * Experiment Tracking CleanUp (#2278) I did a final clean-up of Experiment Tracking on Kedro-viz mostly minor stuff here and there removed dataset_factory resolver implementation as it only affected Exp. Tracking also Experiment Tracking last run data was also originally showed in the flowchart metadata panel, so I removed that besides that some plotly references and other stuff. also quite a few of these files are removing experiment tracking data files from demo-project. * update github-pr-check to github-pr-review to avoid the max file review on PR Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> * ignore value on mergekeeper Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> * Fix typo in ignored CircleCI checks list --------- Signed-off-by: tynandebold <thdebold@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com> Signed-off-by: rashidakanchwala <rashida_kanchwala@mckinsey.com> Co-authored-by: Tynan DeBold <thdebold@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jitendra Gundaniya <38945204+jitu5@users.noreply.github.com> Co-authored-by: Huong Nguyen <huong.nguyen@mckinsey.com> Co-authored-by: rashidakanchwala <37628668+rashidakanchwala@users.noreply.github.com> Co-authored-by: rashidakanchwala <rashida_kanchwala@mckinsey.com>

Huong Nguyen added 3 commits December 2, 2024 21:28

remove session store

11989df

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

remove relateds tests

943699d

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

remove doc about session store

7cd9f79

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

Huongg changed the title ~~Feature/remove session store~~ [DRAFT] Feature/remove session store Dec 2, 2024

Huong Nguyen added 7 commits December 3, 2024 10:01

lint formatting

cc50ab6

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

remove session store from tests

d283fca

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

update docs reference

e083357

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

update test_server

d8abbd3

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

remove sessions store from load_data

2fb3576

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

remove make_db_session_factory

ab66d10

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

remove session store from _load_data_helper

0490962

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

Huongg requested a review from merelcht December 3, 2024 15:24

ravi-kumar-pilla reviewed Dec 3, 2024

View reviewed changes

Huong Nguyen added 7 commits December 3, 2024 16:36

remove database.py entirely

57cba0b

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

remove test graphql folder

3ff2a07

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

remove cy test related to ET tab

4ad56fa

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

Revert changes for .md as it will be done separately

7bf73d7

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

revert contributing.md

ccb3e26

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

Merge remote-tracking branch 'origin' into feature/remove-session-store

60b4e09

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

remove e2e test failed from ET

96488b3

Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

Huongg marked this pull request as ready for review December 4, 2024 13:50

Huongg requested review from rashidakanchwala, astrojuanlu and jitu5 as code owners December 4, 2024 13:50

Huongg changed the title ~~[DRAFT] Feature/remove session store~~ Feature/remove session store Dec 4, 2024

Huongg removed the request for review from astrojuanlu December 4, 2024 13:57

Huongg requested review from SajidAlamQB and removed request for rashidakanchwala December 4, 2024 13:57

jitu5 approved these changes Dec 5, 2024

View reviewed changes

SajidAlamQB approved these changes Dec 5, 2024

View reviewed changes

Huongg changed the base branch from main to feature_branch/remove-experiment-tracking December 6, 2024 10:45

Merge branch 'feature_branch/remove-experiment-tracking' into feature…

cd42a83

…/remove-session-store Signed-off-by: Huong Nguyen <huong.nguyen@mckinsey.com>

Huongg merged commit 7ca0bd0 into feature_branch/remove-experiment-tracking Dec 9, 2024
46 of 56 checks passed

Huongg deleted the feature/remove-session-store branch December 9, 2024 10:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature/remove session store #2219

Feature/remove session store #2219

Uh oh!

Huongg commented Dec 2, 2024 •

edited

Loading

Uh oh!

ravi-kumar-pilla Dec 3, 2024

Uh oh!

ravi-kumar-pilla Dec 3, 2024

Uh oh!

ravi-kumar-pilla Dec 3, 2024 •

edited

Loading

Uh oh!

Huongg Dec 4, 2024

Uh oh!

ravi-kumar-pilla commented Dec 3, 2024

Uh oh!

Huongg commented Dec 4, 2024

Uh oh!

tynandebold commented Dec 5, 2024

Uh oh!

Huongg commented Dec 5, 2024 •

edited

Loading

Uh oh!

jitu5 left a comment

Uh oh!

SajidAlamQB left a comment

Uh oh!

Uh oh!

Uh oh!

Feature/remove session store #2219

Feature/remove session store #2219

Uh oh!

Conversation

Huongg commented Dec 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

ravi-kumar-pilla Dec 3, 2024

Choose a reason for hiding this comment

Uh oh!

ravi-kumar-pilla Dec 3, 2024

Choose a reason for hiding this comment

Uh oh!

ravi-kumar-pilla Dec 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Huongg Dec 4, 2024

Choose a reason for hiding this comment

Uh oh!

ravi-kumar-pilla commented Dec 3, 2024

Uh oh!

Huongg commented Dec 4, 2024

Uh oh!

tynandebold commented Dec 5, 2024

Uh oh!

Huongg commented Dec 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jitu5 left a comment

Choose a reason for hiding this comment

Uh oh!

SajidAlamQB left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Huongg commented Dec 2, 2024 •

edited

Loading

ravi-kumar-pilla Dec 3, 2024 •

edited

Loading

Huongg commented Dec 5, 2024 •

edited

Loading