Create worker to sync basefields #1259

hminsky2002 · 2024-10-16T18:54:16Z

This PR introduces a new worker to the API, which allows an administrator to synchronize BaseField data from a remote service instance to a local one. It heavily follows the patterns already laid out for the bulkUpload task/worker, creating and storing an entity (SyncBaseField) in the database to track the status of a synchronization job. Joyously, this functionality means we no longer need a seed file (see commit 2c8879b).

Closes #756

codecov · 2024-10-16T18:59:37Z

Codecov Report

Attention: Patch coverage is 86.62791% with 23 lines in your changes missing coverage. Please review.

Project coverage is 87.30%. Comparing base (5da8598) to head (f1f2ebb).
Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
src/handlers/baseFieldsCopyTasksHandlers.ts	65.85%	14 Missing ⚠️
src/tasks/copyBaseFields.ts	90.19%	5 Missing ⚠️
...e/operations/baseFields/createOrUpdateBaseField.ts	90.00%	1 Missing ⚠️
...ns/baseFieldsCopyTasks/createBaseFieldsCopyTask.ts	88.88%	1 Missing ⚠️
...ions/baseFieldsCopyTasks/loadBaseFieldsCopyTask.ts	88.88%	1 Missing ⚠️
...ns/baseFieldsCopyTasks/updateBaseFieldsCopyTask.ts	90.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1259      +/-   ##
==========================================
- Coverage   87.36%   87.30%   -0.06%     
==========================================
  Files         199      209      +10     
  Lines        2707     2876     +169     
  Branches      387      408      +21     
==========================================
+ Hits         2365     2511     +146     
- Misses        342      365      +23

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

hminsky2002 · 2024-10-17T04:53:52Z

@slifty @jasonaowen finally got this (mostly) tested! I'll try and get to full coverage before merging but I've taken this out of draft status as I believe this is ready for review, with one caveat:

I think the entity name needs to be changed :/ I got very deep into this before considering that. It's not a particularly intuitive name, and it feels awkward to write out in documentation. I figured I'd wait for your reviews before changing. Ready and willing to refactor everything as soon as we decide on a better name for the entity/task.

slifty

Some thoughts on naming and a few little things

src/__tests__/syncBaseFields.int.test.ts

src/database/migrations/0040-create-sync_basefields.sql

src/database/operations/baseFields/createOrUpdateBaseField.ts

jasonaowen

Awesome work, @hminsky2002!

src/database/migrations/0040-create-sync_basefields.sql

src/tasks/__tests__/processSyncBaseFields.int.test.ts

slifty

Good progress; a fair bit of feedback. If it would help to talk through it let me know.

README.md

src/database/queries/baseFields/createOrUpdateByShortcode.sql

src/database/queries/baseFieldsCopyTasks/updateById.sql

src/tasks/__tests__/processBaseFieldsCopyTask.unit.test.ts

slifty · 2024-11-18T20:52:46Z

src/tasks/processBaseFieldsCopyTask.ts

+		return;
+	}
+
+	try {


Am I missing a test? Codecov is saying that all of these lines are covered... but I'm not seeing any tests that actually run anything.

src/tasks/processBaseFieldsCopyTask.ts

slifty · 2024-11-18T21:16:13Z

src/tasks/processBaseFieldsCopyTask.ts

+						}
+						resolve(parsedData);
+					} catch (err) {
+						reject(


Will reject cause it to try again? Do we want it to try again if the format is unexpected?

My understanding is that on promise rejection, the copyBasefields function that calls the fetch intercepts the failure and updates the status of the task, which I believe means that it will not get called again (which I think is what we want, so long as it is logged!)

src/types/BaseField.ts

jasonaowen

This is the first major PR linted with SQLFluff, so we have a few things to shake out!

src/database/migrations/0042-create-base_fields_copy_tasks.sql

src/database/queries/baseFieldsCopyTasks/selectById.sql

hminsky2002 · 2024-12-13T19:34:14Z

@slifty re-requesting review here (and apologies for letting yours hang for so long) -- agreed with all your notes regarding language of variables and the status_updating being handled by the database itself (very cool!) -- major thing left to discuss is nock with fetch as i mentioned above, interested to hear your take on the matter!

mikicho · 2024-12-18T20:07:21Z

@hminsky2002, you can try Nock's beta for native fetch support. We are about to release it (v14) soon :)

jasonaowen

This is looking good, @hminsky2002! Thanks for working on this.

src/database/queries/baseFieldsCopyTasks/selectWithPagination.sql

src/tasks/copyBaseFields.ts

This looks good and Jason has reviewed the latest closely!

This commit adds an upsert query and operation for basefields, following the pattern laid out for basefield localizations. We dont want the synchronization to be destructive, so we update on shortcode.

This commit creates the new entity 'BaseFieldsCopyTask', along with the necessary database operations to access and update them. BaseFieldsCopyTasks, like BulkUploadTasks, are intended to serve as user-accessible entities that log the status of a basefield copy job between a local and remote pdc instance. In order to use nock, our current http mocking framework, for testing, we needed to use an alternative to the default node fetch package, as that is incompatible with nock. We decided in the PR to add the node-fetch package, which is compatible with nock. Notably we added it at an older version, which per the node-fetch documentation is necessary for commonJs compatiblity (https://github.com/node-fetch/node-fetch#commonjs)

hminsky2002 force-pushed the 756-create-worker-to-sync-basefields branch 2 times, most recently from de7e177 to daba11e Compare October 17, 2024 04:30

hminsky2002 marked this pull request as ready for review October 17, 2024 04:48

hminsky2002 requested review from slifty and jasonaowen October 17, 2024 04:48

hminsky2002 force-pushed the 756-create-worker-to-sync-basefields branch 2 times, most recently from 1f27e03 to 6254d60 Compare October 17, 2024 05:02

slifty requested changes Oct 22, 2024

View reviewed changes

jasonaowen requested changes Oct 24, 2024

View reviewed changes

hminsky2002 force-pushed the 756-create-worker-to-sync-basefields branch from 6254d60 to 9704bee Compare October 25, 2024 02:10

This was referenced Nov 4, 2024

Refactor BulkUpload entity into BulkUploadTask #1305

Closed

Refactor bulk upload to bulk upload task #1312

Merged

hminsky2002 force-pushed the 756-create-worker-to-sync-basefields branch 10 times, most recently from 4076491 to 4b20c24 Compare November 18, 2024 03:47

hminsky2002 requested review from slifty and jasonaowen November 18, 2024 03:49

slifty requested changes Nov 18, 2024

View reviewed changes

hminsky2002 mentioned this pull request Dec 2, 2024

Refactor AddBulkUploadTaskJob to AddBulkUploadTask #1346

Closed

hminsky2002 force-pushed the 756-create-worker-to-sync-basefields branch 2 times, most recently from ecb1b38 to a2fd57a Compare December 2, 2024 01:12

jasonaowen reviewed Dec 10, 2024

View reviewed changes

jasonaowen mentioned this pull request Dec 12, 2024

Use lowercase for SQL function and type names #1380

Merged

jasonaowen force-pushed the 756-create-worker-to-sync-basefields branch from e0873cc to f3256a6 Compare December 12, 2024 23:06

hminsky2002 force-pushed the 756-create-worker-to-sync-basefields branch 4 times, most recently from c4ead92 to 3dbd926 Compare December 13, 2024 19:11

hminsky2002 requested review from jasonaowen and slifty December 13, 2024 19:32

hminsky2002 force-pushed the 756-create-worker-to-sync-basefields branch 4 times, most recently from 21404ab to aab573d Compare December 18, 2024 07:46

hminsky2002 force-pushed the 756-create-worker-to-sync-basefields branch from aab573d to ca4f9f0 Compare December 19, 2024 21:22

jasonaowen approved these changes Dec 19, 2024

View reviewed changes

src/database/queries/baseFieldsCopyTasks/selectWithPagination.sql Outdated Show resolved Hide resolved

src/tasks/copyBaseFields.ts Outdated Show resolved Hide resolved

src/tasks/copyBaseFields.ts Show resolved Hide resolved

hminsky2002 force-pushed the 756-create-worker-to-sync-basefields branch 3 times, most recently from 54fa64e to 0b9f4a5 Compare January 7, 2025 17:33

hminsky2002 added 4 commits January 7, 2025 12:38

Add upsert operation for basefields

8d64d22

This commit adds an upsert query and operation for basefields, following the pattern laid out for basefield localizations. We dont want the synchronization to be destructive, so we update on shortcode.

Add BaseFieldCopyTasks handler and task routing

cb6d43c

remove seed file and references to seed file

f1f2ebb

hminsky2002 force-pushed the 756-create-worker-to-sync-basefields branch from 0b9f4a5 to f1f2ebb Compare January 7, 2025 17:38

hminsky2002 merged commit 26f8d6f into main Jan 7, 2025
9 of 11 checks passed

hminsky2002 deleted the 756-create-worker-to-sync-basefields branch January 7, 2025 17:44

hminsky2002 mentioned this pull request Jan 14, 2025

Remove Seed file #1441

Closed

bickelj mentioned this pull request Jun 10, 2025

Add sensitivityClassification #1686

Merged

+              		return;
+              	}
+              	try {

Create worker to sync basefields #1259

Create worker to sync basefields #1259

Uh oh!

Conversation

hminsky2002 commented Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hminsky2002 commented Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slifty left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jasonaowen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

slifty left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

slifty Nov 18, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

slifty Nov 18, 2024

Choose a reason for hiding this comment

Uh oh!

hminsky2002 Dec 2, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jasonaowen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hminsky2002 commented Dec 13, 2024

Uh oh!

mikicho commented Dec 18, 2024

Uh oh!

jasonaowen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hminsky2002 commented Oct 16, 2024 •

edited

Loading

codecov bot commented Oct 16, 2024 •

edited

Loading

hminsky2002 commented Oct 17, 2024 •

edited

Loading