Cleanup (Server) #3 - Tasks: A-1, 3 #988
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Refer to details in cleanup issue: Task A-6: e-mission/e-mission-docs#1082 (comment)
Refer to details in cleanup issue: Task A-2: e-mission/e-mission-docs#1082 (comment) Storing server tag as well so that artifacts are not needed. Can also remove image tag passed as input in Workflow dispatch POST request. Workflow input also removed in dashboard workflows For now not removing artifacts until the internal script is updated to handle this change. ---- Task A-8: Prefixing branch name to the docker tag along with the date. In the internal script we will not need to maintain the different branch lists as the images will be completely tagged in the external workflows themselves. We can simply use the tags without modifications then. For now, not prefixing the tag to the artifact since we will be removing the artifact anyways. And current internal script works with artifacts. Once I update the internal script, will come back and remove artifacts.
…ally Refer to issue comment for details: Task A-7: e-mission/e-mission-docs#1082 (comment) The certificates are relevant to our internal AWS configuration and not needed externally. They can be present externally too without having any major effect. But removing them helps keeping the base image clean. Additionally, anyone working with the code can customize with their own certificates if needed or adopt an approach which doesn't even need certificates in the first place.
Internal script updated as well. Internal PR must be merged as well once these external PR changes merged.
Storing a reusable workflow in the e-mission-server repo. Can decide where to place it in a central location. https://docs.github.com/en/actions/sharing-automations/reusing-workflows It essentially works like a function call in normal programming. The advantage is that we have no repeated code the image build process. All the other repos (join, admin-dash, public-dash) reuse the same workflow file. Additionally, on for future GitHub actions, workflow file related changes, will no longer need to have 3 additional PRs for each repo (join, admin-dash, public-dash). Can simply modify the reusable workflow file as this is the core “function” workflow that is being called. I have added conditional checks that check for the repo name in the reusable workflow file that determine which statements to execute depending on for which repo the workflow is running. This is used for both push events specific to a repo as well as for the workflow dispatch events triggered on pushes to server repo.
Approach: Converted test-with-docker and test-with-manual-install to reusable workflows. Added them as jobs in image_build_push workflow. Build job in this workflow needs these test jobs as dependencies. ------- Test Workflow run https://github.com/MukuFlash03/e-mission-server/actions/runs/11096277197/job/30825929886 ----- Notes Github actions didn't have out of the box solution for running a workflow based on results of multiple workflows where ALL workflows must have completed successfully. We need this since both the test-with-docker and test-with-manual-install must pass. So this needs an "AND" logic. "workflow_run" is there but this triggers the dependent workflow when either of the workflow dependencies defined as prerequisites are completed. So this has an "OR" logic. https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#workflow_run ---- Found an alternative suggestion here: https://stackoverflow.com/a/75597437 This suggests converting the pre-requisite workflows into reusable workflows. These workflows can then be called in jobs in the workflow that needs these workflows to be run before. Finally, these jobs can be added as dependencies for the requisite job. In our scenario, two new jobs are added to the image_build_push.yml for each of the two tests environments. These will run parallelly. Then in the build image job, these jobs are added in the "needs" field, indicating that these jobs must pass successfully before running the build job. ------- Also corrected the branch in reusable workflow for fetching latest server image tag. Need to update in the currently open PRs as well. ------ Also, removed the `push` trigger from the two `test` workflows since the `image_build_push` workflow would also be triggered on a `push` event which in its workflow triggers these two test workflows. Thus if not removed, each test workflow would run twice.
1. Modified server to use reusable workflow as well - Lots of repeated code in this workflow as well. 2. Added tags as push event triggers - Github documentation for pattern matching - https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#patterns-to-match-branches-and-tags - v[0-9]+.[0-9]+.[0-9]+ 3. Using github context data to select correct branch “git push” command failed in reusable workflow since it wasn’t listing any branch and by default it picked up the tag version and not any branch name. Used ${{ github.event.base_ref }} that has the source branch from which the release tag was created. - https://github.com/orgs/community/discussions/27154 - https://github.com/orgs/community/discussions/26243 Needed to extract only branch name to ignore “ref/heads/“ Added a separate job that extracts this using bash commands - https://stackoverflow.com/questions/16623835/remove-a-fixed-prefix-suffix-from-a-string-in-bash ——— 4. Server workflow runs on both push:branches and push:tags events. This is because we want tests to run before push. Using “github.ref_type” for checking whether event type was “branch” or “tag” - Documentation - https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/accessing-contextual-information-about-workflow-runs Server run was failing: "git push” needs the branch name - https://github.com/MukuFlash03/e-mission-server/actions/runs/11134355511/job/30942421505 - Add branch name in reusable workflow The error message in the run logs suggest using this: - https://github.com/MukuFlash03/e-mission-server/actions/runs/11134355511/job/30942421505 - $ git push origin HEAD:<remote-branch> ———— 5. Using github.ref_type to check “branch” or “tag” and select correct branch When server release is created, it triggers dispatch to other workflows. However these don’t have “github.ref_type” = “tag” in this case but instead “github.ref_type” = “branch”. This case github.event.base_ref value to be empty, hence branch name extraction didn’t work. I’m using the ref_type to pass either GITHUB_REF or the extraction logic using github.event.base_ref. This is done in the workflows that call reusable workflow. The check should be fine since ref_type has only two values: branch or tag - https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/accessing-contextual-information-about-workflow-runs
…nd not push Task A-3: Image build/push on release instead of on every push 1. Re-building on releases instead of tags - Earlier I had set the trigger event as tags but this would mean any time we create a new tag the workflow would be triggered. - Instead, we are actually creating releases and not tags through the UI. - Hence change the trigger event to releases, specifically “published” releases. ------ 2. Both Tests and Build will happen for push events - Tests: We need tests to confirm everything works before merging. - Build: Similarly, even though tests might pass, the image build could fail due to issues in Dockerfile or any files used in Dockerfile. - Hence to ensure that the final image that is pushed is correctly built, and we can have a correct version released, running both tests and build for push events. ------ 3. Dispatch only runs on releases identified by ref_type == tag. - Dispatch depends on build and since we now have build running for both push and release events, dispatch would also run for both. - Hence restricting dispatch to run only on release events by using the ref_type check which has the value “tag” for releases and “branch” for pushes on any branch. ------- 4. Additional version increment steps + Creating incremented version tags and releases in workflow - Manually creating new release through UI would trigger the workflows for all the repos and the new tag would be updated to the tag version set in the UI. This would be updated in the .env files as well. - But for the workflow_dispatch as well, we need to handle creating a new tag and release for the dependent cascading images in the dashboard repositories. - For this, I’m fetching the latest and 2nd latest server releases and checking for the type of SemVer release (major/minor/patch). Based on the type, similar version increment would occur for the dependent images. - Depending on the event type (release / workflow_dispatch), either the tag created through the UI or the incremented tag version would be chosen as the new tag, respectively for each of the two trigger event types. - This new tag is also written to the .env file. - Finally, a new release is created based on this newly created tag in the workflow. ---------------------- Task A-1: Switch to storing images in GHCR 1. Changes to Dockerfile - Needed to add LABEL layer to ensure that the docker image is connected to the correct repository. - For dashboard images that are dependent on the server image, changed to pull from GHCR. 2. Workflow file - Login action used to login to GHCR. - Building, Renaming, Pushing images with GHCR tag. - Appropriate version tag names used depending on whether trigger event was release or workflow_dispatch.
14 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Third set of cleanup changes for the redesign cleanup issue
These should be merged only after the second set of PRs are merged.
Includes changes for tasks:
Task A-1: Switching to storing images in GHCR
Task A-3: Image build/push on release instead of on every push