Implement a message queue between gateway and workflow service for improved reliability #7559

rikukissa · 2024-08-30T15:27:56Z

To-do

Implement a new endpoint for country config to reject a registration (like confirm registration). The current synchronous flow where the /event-registration can return anything other than 2** status to indicate the request got accepted needs to be refactored. Endpoint needs to confirm the message was received and then call another endpoint in core to let core know if the record got rejected or accepted.
- Done here: feat(message-queue): allow confirming and rejecting record externally #7674
Allow configuring a port, username and password for Redis connection. Let services pass a full RedisOptions object inside useRecordQueue
If the services are restarted, the token expires in queue the same issue can happen if a message is in the queue for too long and the user token expires. Instead of us storing the token in the message queue payloads, let's instead store the initiating user's id and roll a new token in the code the message queue triggers.
Fix conflicts on this branch
Investigate the failing tests: should they be removed or rewritten?
- Fix the tests

Dependencies

Hide all internal services behind one url #6279
- This needs to be done so that we can implement an endpoint to the auth service to request a token for a given user id without the endpoint being exposed to public internet like we have all of them today.

Technical tests

How does Redis queue fair with large attachments?
How does both the queue and frontend behave when you create multiple drafts offline and then come back online

Tips

Run this to see a local GUI for your message queue items

docker run -p 3003:3000 -e REDIS_HOST=host.docker.internal  deadly0/bull-board

github-actions · 2024-08-30T15:28:11Z

Oops! Looks like you forgot to update the changelog. When updating CHANGELOG.md, please consider the following:

Changelog is read by country implementors who might not always be familiar with all technical details of OpenCRVS. Keep language high-level, user friendly and avoid technical references to internals.
Answer "What's new?", "Why was the change made?" and "Why should I care?" for each change.
If it's a breaking change, include a migration guide answering "What do I need to do to upgrade?".

…e queue between gateway and workflow service

rikukissa · 2024-09-17T10:49:41Z

packages/commons/src/fhir/task.ts

 > & {
  lastModified: string
  status: 'ready' | 'requested' | 'draft' | 'accepted' | 'rejected'
  extension: Array<Extension>
+  focus: {


Just a random type improvement for Task as it came up as part of the task. Don't think there is ever a case where a Task wouldn't reference a composition in focus.reference

…s-6208

rikukissa · 2024-09-17T10:58:04Z

packages/commons/src/message-queue/index.ts

+ * Copyright (C) The OpenCRVS Authors located at https://github.com/opencrvs/opencrvs-core/blob/master/AUTHORS.
+ */
+
+export * from './record'


This is our new message queue module :) All posting to and listening from the queues should happen through this module in commons. This ensures that the type chain doesn't break in the code.

rikukissa · 2024-09-17T11:10:13Z

packages/gateway/src/workflow/index.ts

-    trackingId: string
-    isPotentiallyDuplicate: boolean
-  }>('POST', '/create-record', authHeader, { record, event })
+  if (inScope(authHeader, [USER_SCOPE.REGISTER])) {


Remember how the "create record" handler is basically 3 operations in one? I split it and lifted the detection to this level. Ideally the client would call right graphql handler from the start

nice one 👍

rikukissa · 2024-09-17T11:15:46Z

packages/metrics/package.json

@@ -25,6 +25,7 @@
    "csv-stringify": "^5.3.4",
    "csv-writer": "^1.6.0",
    "date-fns": "^2.28.0",
+    "envalid": "^8.0.0",


This was added, rest are just reordered

rikukissa · 2024-09-17T11:59:05Z

packages/metrics/src/api.ts

  method = 'GET',
  body?: string
 ) {
  const url = [fhirUrl.replace(/\/$/, ''), suffix].join('/')
  return fetch(url, {
    method,
    headers: {
-      'Content-Type': 'application/fhir+json',
-      ...authHeader
+      'Content-Type': 'application/fhir+json'


Ok, here starts some not-very-major but many-in-numbers changes start. I removed the way we send auth tokens to Hearth as it doesn't use it. for anything. More of a risk than a benefit if nothing in the receiving end uses it

Only token removals in this file.

rikukissa · 2024-09-17T12:01:46Z

packages/metrics/src/features/audit/handler.ts

    const payload = request.payload as IUserAuditBody
+
+    const transactionId = payload.transactionId


Transaction IDs are used to determine if an operation has already been performed. A service can for instance check if there is already a database row with a certain ID. If yes, then it wont write it again

rikukissa · 2024-09-17T12:02:42Z

packages/metrics/src/features/audit/handler.ts

@@ -50,7 +55,8 @@ export async function newAuditHandler(
        payload.additionalData
      )
    )
-    await writePoints(points)
+
+    await writePoints(points, transactionId)


writePoints function is now in charge of never writing duplicates even if the handler is called multiple times (given transaction id stays the same)

rikukissa · 2024-09-17T12:05:29Z

packages/metrics/src/features/export/handler.ts

@@ -61,9 +61,7 @@ export async function monthlyExportHandler(
  if (monthlyMetrics.genderBasisMetrics) {
