-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Backend state migration #1101
Open
Peedee2002
wants to merge
139
commits into
dev
Choose a base branch
from
331-staging
base: dev
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Backend state migration #1101
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…Circles into CIRCLES-331/user-data-to-backend
…irst tab load (#1144) * downsized tab loading status for course tab loads. * added reusable spinner wrapper style (small and large) and removed element styling
* added UNSW course outline link under the UNSW course handbookinside the courseAttributes.tsx * fixing the link where the link would track the term of course when its first avaliable * adding the termMapping for the TermMode inside the course outline link * adding the feature where it would go back to the last yera when the current term has not been reached * reverting the file * adding the cours eattribtues from friday * fixing if the terms offered is 0 * considering hte teaching period for the summer term * adding for hte teaching period * repushing fixing typo * fixing the style removing the import and also combing T and updated Term
* changed from loading to empty grid on no course select * fixed conditional statement for loading * please let this one be okay * add data to coursesInfoFlipped * refactor outside function * make func name descriptive * fix linting issues --------- Co-authored-by: Deki Lhamu <z5268870@student.unsw.edu.au>
* degree wizard * restrict start date * edits to start date in settings menu * Update frontend/src/pages/DegreeWizard/YearStep/YearStep.tsx Co-authored-by: olli <80164276+ollibowers@users.noreply.github.com> * Update YearStep.tsx --------- Co-authored-by: olli <80164276+ollibowers@users.noreply.github.com>
* wip: setup redirecting and dummy token validation * wip: i believe i can do the full auth flow now * wip: playing around with oidc * wip: still working on authentication, oidc engulfs my mind this is fun but i have spent hours on this * wip: working on integrating all parts together * wip: started working on session middleware * wip: refactored requests helpers a little bit * wip: got some semblence of sessions working * feat: setup better errors for oidc and session * wip: playing with cookies * wip: ahhhhhhh * wip: started getting somewhere with token pair flow * wip: hooked up oidc checking/refreshing to the identity and userinfo * wip: worked on identity fetching and more route protecting * wip: made route to generate auth url * wip: login url is now generated by backend * wip: now verifying at_hash * wip: setup some dummy setup checking * fix: remove levenstein * wip: fixup after merge * wip: started refactor for new auth flow * wip: redesigned the session interface * wip: brainstorming new mongo and redis structure * wip: changed sids to uuids * wip: refactored session interface to check replay attack earlier * wip: started rewriting auth routes - identity * wip: wrote login route * wip: rewrote rest of the main auth routes * wip: the rewrite base flow works omg * fix: now deletes state cookie if invalid auth code given * wip: better error handling for logout * wip: address most todos in oidc/requests.py * wip: setup redis docker * wip: hooked up the session token part to redis * wip: started to refactor users db * wip: added session related collections * chore: fix up packages after rebase * wip: converted over refresh token storage to mongo * wip: finished converting all to mongo (first step) * fix: didnt realise pagination happened on ft search ugh i hate this * feat: ttl on redis session tokens wooo * feat: indexes on session collections mongo * feat: converted sids back to uuids * feat: fixed up validation for new user storage * wip: started integrating guest sessions * wip: started to refactor storage interface * wip: culled some interface functions * feat: GUEST SESSIONS (not in user layer yet) * wip: fixed up model props in anticipation for user overhall * wip: added user validation schema and collection typing * feat: created new user database helpers * wip: remove uid from models temporarily and fix pylint * wip: added new user storage setup on session creation * wip: db refactor - refactored out the redis connection * wip: db refactor - deleted storage.py and created all da helper files * wip: db refactor - remove cringe db helper prefixes * wip: db refactor - moved out mongo conns into new file * wip: db refactor - no more database.py * wip: db refactor - started combining init-mongo and init-sessionsdb * wip: db refactor - combined init sessionsdb * fix: remove nanoid and bring back old package-lock * wip: user routes refactor - underwrote old set and get * wip: user routes refactor - hooked up guest login and started with fixing edge cases of underwrite * wip: user routes refactor - fixed bugs with degree wizard and db helpers update success checks * wip: user routes refactor - fix for None marks * wip: frontend token refactor - re-setup guest login and id providing and refreshing * wip: frontend token refactor - created new redux slice for identity * wip: frontend token refactor - fixed any straggling old redux usage * wip: frontend token refactor - token param - addToUnplanned & removeCourse * wip: frontend token refactor - token param - setPlanned & setUnplanned & unschedule & unscheduleAll & ignoreFromProg * wip: frontend token refactor - token param - removeAll & validateTermPlanner * wip: frontend token refactor - token param - setupDegree & resetDegree * wip: frontend token refactor - token param - search & updateMark & validateCTF & toggleLocked * wip: frontend token refactor - token param - getUser * wip: frontend token refactor - token param - getUserDegree * wip: frontend token refactor - token param - getUserPlanner & getUserCourses & setIsComplete * wip: frontend token refactor - completely remove getToken * wip: frontend token refactor - setup identity provider and playing around with different refresh methods * wip: frontend token refactor - reworked token state checking and RequireToken * wip: frontend token refactor - Wrapped routes in RequiredToken and PreventToken * wip: frontend token refactor - fixed setup and reset on frontend * wip: frontend token refactor - fixed PreventToken temporarily * wip: frontend token refactor - remove Auth.tsx * wip: frontend token refactor - add back pagetemplate to Login.tsx * wip: frontend token refactor - create Logout page and fix up some query client invalidation * wip: frontend token refactor - fix bug with refreshing again * wip: fix up models after pydantic upgrade * fix: unplanned courses have null for unplanned * wip: moved all tokens to be passed via headers * feat: create connect function for mongodb conn * feat: replaced nodemon with uvicorn reload for better control of what reloads * feat: effectively removed init-mongo * fix: removed old collections * feat: new straight to uid route dependency * wip: minor route rename * fix: get rid of field since this is now inferred by the shape of the user * fix: added WWW-Authenticate headers to 401 errors * feat: delete all guest data on logout * feat: added dev flag to runserver.py so production dont reload * chore: relabeled all my todos lol * chore: forgor some * feat: setup secure cookies * fix: search bar quick add buttons not updating * feat: redis password * fix: fixed up login success page (i think idk auth is down) * chore: relabelled todos so i actually know when i need to do what * fix: dict() -> model_dump() * fix: removed the token state route in place for isSetup * wip: changed the isSetup error handling to use throwOnError * wip: added userinfo validation p1 * wip: fixed logout redirect and pondered on things * wip: playing around with diff refresh methods * wip: removed whacky thunks for refreshing * wip: change secure cookie prefix and trying out samesite strict * deps: fix after merge * fix: cookies now get deleted on secure mode * fix: rename session errors * wip: investigated sid and logout again * wip: more todo culling and cleanup * wip: converting over oidc config to be dynamic * fix: summer term off by default * fix: move oidc config into helper (kinda mid sol rn) * fix: changed logout to be post * wip: started working out the redirect behaviour of isSetup 401 * fix: useToken now can actually hit error boundary * fix: temporarily solved the redirection 401 conundrum * fix: delete init-mongo from docker-compose * fix: removed some debug token help * feat: made logout a bit more robust i guess * fix: forgot to remove a debug error in landingpage im dumb as hell * fix: oop forgot some more debug comments in IdentityProvider * fix: made the environment capture more explicit for run_app.py * fix: cleanup auth.py logout route a lil * fix: bug with double initial refresh and csesoc logins * fix: changed logout to use hard navigation for data clearing * fix: removed token playground * fix: deleted some unneeded user db helpers * feat: rewrote mongo setup a bit so that it doesnt drop users by default * feat: moved collection names to constants and renamed the dyn cols * feat: redis now gets cleared on startup * fix: made some mongo setup funcs exportable for testing purposes * fix: added redis reset to test clear helper * fix: tests can run locally now given you have env files * feat: backend can now run without fedauth credentials * update CI for testing with new stuff * update documentation * fix readme * fix: cleaned up various return values and files after PR review feedback - fixed return values of delete session and refresh token helpers - removed ideas.md file - changed oidc error string method to use tabs instead of spaces - renamed state cookie and its ttl constant * fix: droppped 'New' naming from collection variables * bump random thingo to remove high vuln * fix: explored better marker typing for models, found out pydantic is broken * fix: major issue of spelling raised by zax-xyz Co-authored-by: Michael Vo <zax@zaxu.xyz> * fix: cleaned up identity slice reducers and thrown errors on frontend As per Michael feedback! * fix: removed all the spreads on the withAuthorization, as per Michael feedback :) * fix: removed suppressed and mostRecentPastTerm to align with #1158 * feat: added new cli arg for deleting user data aswell, updated ci backend container to mirror prod but use this Also finally deleted init-mongo.dockerfile and init-database.py * fix: updated ci to use env script * fix: mypy issues with unsupported typing
* wip: re enable pylint for 331-staging * wip: trying to disable the frontend tests and adding linting and type checks * fix: forgor working directory * fix: install dotenv for frontend * wip: add workplace extensions and LF the .gitignore * wip: testing out a composite action * fix: forgor shell from my composite action * wip: fix for pip install dotenv fail * wip: i might be dumb * wip: more testing, double python usage * wip: added eslint job to test if it all works * fix: wrong lts version * wip: added first few jobs * wip: added the backend jobs * fix: backend tests failed, added wait for backend container * chore: removed the old pylinter file * fix: remove shell from sleep * fix: pylint - trailing-whitespace, trailing-newlines, missing-final-newline * fix: pylint - ungrouped-imports, wrong-import-position, reimproted, wrong-import-order Also changed run_app.py checks to no longer require nodemon. * fix: style - too-many-locals, too-many-function-args, cell-var-from-loop Mostly just added scope pylint disables because I am not refactoring this code. * fix: pylint - possibly-used-before-assignment, unused-import * fix: pylint - unused-variable * fix: pylint - multiple-statements, redefined-outer-name, no-else-return, consider-using-enumerate * fix: pylint - bare-except, use-dict-literal * fix: actual troll * fix: pylint - subprocess-run-check, consider-using-sys-exit * fix: pylint - consider-using-with, abstract-method, inconsistent-return-statements, unspecified-encoding * wip: disable the unit tests for now * fix: pylint - too-many-nested-blocks, too-many-return-statements, too-many-branches, too-many-public-methods Most of this was just disables, as I believe these were used fairly and solemnly. * fix: pylint - empty-docstring (and some new todo comments) * feat: added healthcheck for backend containers to be used in ci, created a /ping route to achieve this * feat: docker build job now requires all other jobs, as pointless check otherwise * fix: re-disabled the backend tests * chore: add 2024 team.md and update previous years for missing members * chore: add 2022 luke github username * fix: remove test jobs from docker job needs * fix: pylint - remove redundant and leftover pylint disables * fix: eslint - allow console.error and non-default exports in the api directory (until error reporting is standardised) * wip: eslint - removed any redundant eslint-disabled * chore: added some todo comments for backend type removal * fix: eslint - removed the eslint-disable from TermPlanner and cleaned it up a lot * fix: changed courseQueries.isPending to be isPlaceholderData since pending is always false there * fix: eslint - removed a bunch of unnecessary eslint disables * fix: eslint - moved out two axios calls as to resolve some eslint issues in their users * fix: refactored graph to use unlocked course state for coloring as .unlocked doesnt exist on user courses
* Fix all token related backend tests * fix some tests * fix a couple of more tests * fix a bunch more tests * all backend tests pass * Refactor get_token * Delete debugging print statement * fix: added timeout to test utility request --------- Co-authored-by: Peter Derias <peter.derias@gmail.com>
* feat: re-enable backend tests in ci * hotfix: moved the env setup in test utilities above the connections * fix: moved out /courses/getCourse, /courses/getPathFrom and /courses/courseChildren Also deleted a unused helper. * fix: moved out timetable request * fix: moved out coursesUnlockedWhenTaken, and consolidated courses.ts into course.ts * fix: move out /programs/getPrograms and fold programApi.ts into programsApi.ts * fix: move specialisation api calls into specsApi.ts, upgrading useEffects to useQuerys in degree wizard. * fix: move /user/updateDegreeLength, /user/toggleSummerTerm, /user/updateStartYear, /planner/toggleTermLocked into api folder * fix: moved out ctf axios (lol) and finished last few others * chore: rename some fe api request files and shuffled some functions * fix: final useEffect in ProgressionChecker * fix: move out unwrapSettled api helper * fix: moved Prereq tree to use queries Also explored size fixing, left a TODO comment with my findings, so we can revisit later.
* Abstract settings slice behind useSettings * Separate backend and frontend settings * Fix linting * Reimplement hidden years * Fix reset button * Fix set_user * Fix incorrect import
* Add support for removing marks from a course * Fix error message * Fix stale validation for unplanned courses * fix: added some missed invalidations of courses that prevented accurate context menu --------- Co-authored-by: ollibowers <80164276+ollibowers@users.noreply.github.com>
* adding the lgoout button inline with the welcome to Circles * restoring the git statuses * fix: move logout button in line with subtitle
…uce coupling (#1189) * refactor(utility): removed reliance on get_course route function, created get_course_details in utility.py * refactor(utility): created term offering helpers to remove reliance on terms_offered route function * refactor(utility): created specialisation getter, refactored setupDegreeWizard to not rely on route functions * refactor(utility): created get_program_structure utility and moved out all dependencies from program routers * refactor(utility): moved out regex_search to utility.py * wip: move out user helpers into their own file * refactor(utility): moved out logic for and some graph helpers into utility.py * fix: switched up setup_degree_wizard so we can get a bit better type inference * fix: new line at end of user.py * fix: remove unused CODE_MAPPING cache constant * refactor: routers folder structure - move middleware.py into sessions utility folder * refactor: routers folder - rename auth_utility folder, move utility.py into it and rename * refactor: routers folder - move manual_fixes into utility folder * fix: remove my todo list
* updated scraper to use new api and scrape postgrad items * filter in only undergrad courses * remove debug prints * fix pylint, remove old (unused) create_payload function * fix pylint again * fix pylint fr this time * rerun format and process courses * scrape only ugrd prgms and spcs * re pickle conditions and check certain courses exist before changing condition * fix be tests
* Replace degree wizard dropdown with antd select * Restore original degree fuzzy search * Switch to antd select for specialisation step * Animate div for specialisations * Improve degree fuzzy search and validation * Fix review comments
…#1195) * fix: change eslint rules so frontend can compile with unused variables * wip: created mirrored prepareUserPayload backend helper and converted * fix: tokenified the route * fix: coursesUnlockedWhenTaken and getAllUnlocked tests now use tokens * remove prepareUserPayload from frontend * feat: created function, and finally removed old UserData model * fix: unselect tests now use token * added note into unselectCourse route * fix: aligned mark mappings to the WAM spec * wip: move out convert_to_planner_data * feat: removed model and reworked route * fix: convert ctf to fully use user Storage * feat: convert /validateTermPlanner to no longer use old models * fix: update frontend mark parsing to match backend * fix: re-enable circular imports pylint and remove type keyword * fix: remove potentially bad load_dotenv * fix: remove unused props for CourseDescriptionPanel and CourseMenu after tokenifying last routes
… yet (#1196) * infer years forward * work for degree after LIVE_YEAR Co-authored-by: olli <80164276+ollibowers@users.noreply.github.com> --------- Co-authored-by: olli <80164276+ollibowers@users.noreply.github.com>
* Add JSON import and export support * Clean-up unused code * Move degree validation to helper function * Fix CI * Fix tests * fix test to reflect a comp program --------- Co-authored-by: Martin <martin.knezevic1@gmail.com>
* Add support for Advanced Computer Science degree * Add support for City Planning (Honours) * Fix tests for City Planning (Honours) * Roll back frontend changes to degree wizard
* rescrape program and specialisation for 2024 * small fix for specs processing, live year to 2025, scrape 2025 * run the rest of the processors for 2025 * update archive * update live year on frontend * update failing tests to reflect new course requirements and pass
* Add local storage migration support * Add automatic migration via query parameters * Strengthen import/export validation * Update migration modal message * Make CSESoc migration more sane
* fix ethics by making prereq preprocess pattern recognise it * update test
…inability (#1205) * feat: main three user queries abstracted * fix: bad imports for new use queries * wip: doing removeCourseMutation * wip: refactored hook creation and added addToUnplanned and removeCourse * feat: updateMark, removeAllCourses, and resetUser hooks... also fixes type no-arg mutations * feat: toggleLockedTerm, toggleSummer, updateStart and updateLength * feat: added support for queryClient and toggleShowMarks, hideYears, showYears hooks * feat: final few mutations * fix: removed user/mutation imports * wip: created most hooks, working on static hooks * Add more static queries * wip: finished all the static hooks * wip: temporary fixed non-null assertion issues * fix: replace combo extended info query with 3 separate queries * feat: applied sensible default settings for queries * fix: refactored graphical selector slightly to finally fix bubble highlighting * fix: final to keys * fix: extracted out the default options --------- Co-authored-by: Lucas <me@lhvy.dev>
…ed leeway to OIDC token validation (#1206)
ChinoGoblino
had a problem deploying
to
production
November 17, 2024 05:27
— with
GitHub Actions
Failure
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The things I'm aware need to be done:
Feel free to add whatever you need too @ollibowers @martanman