CUMULUS-3960: Updated PostToCmr task to be able to republish granules #3906

jennyhliu · 2025-01-21T23:37:46Z

Summary: Summary of changes

Addresses CUMULUS-3960: Update CMR metadata with new collection info

Changes

Updated PostToCmr task to be able to republish granules

Testing

Verified that PostToCmr can republish granule:

Added a workflow with the RepublishTest:

    "RepublishTest": {
    "Parameters": {
      "cma": {
        "event.$": "$",
        "ReplaceConfig": {
          "FullMessage": true
        },
        "task_config": {
          "bucket": "{$.meta.buckets.internal.name}",
          "stack": "{$.meta.stack}",
          "cmr": "{$.meta.cmr}",
          "launchpad": "{$.meta.launchpad}",
          "etags": "{$.meta.file_etags}",
          "concurrency": 10,
          "republish": true
        }
      }
    },
    "Type": "Task",
    "Resource": "${post_to_cmr_task_arn}",
    "Next": "BackupGranulesToLzards",
    "Retry": [
      {
        "ErrorEquals": [
          "Lambda.ServiceException",
          "Lambda.AWSLambdaException",
          "Lambda.SdkClientException"
        ],
        "IntervalSeconds": 2,
        "MaxAttempts": 6,
        "BackoffRate": 2
      }
    ]
  },

Updated spec/parallel/orca/OrcaBackupAndRecoverySpec.js to ingest a granule only (xdescribe any tests except S3 Ingest Granules, comment out the cleanup code)
Got the output cumulus message in s3 from IngestAndPublishGranuleWithOrca workflow execution.
Manually updated .cmr.xml in s3 to have collection ShortName MOD09GQ-> MOD09GQ-AZ
Ran the workflow with the cumulus message we got from IngestAndPublishGranuleWithOrca with etags removed. We can update etags as well.
Verified CMR granule has updated collection ShortName: https://cmr.uat.earthdata.nasa.gov/search/granules.umm_json?provider=CUMULUS&granule_ur=MOD09GQ.A5513498.Nji_76.006.6698559176437
Note: The cmr concept-id remains the same in my test

PR Checklist

Update CHANGELOG
Unit tests
Ad-hoc testing - Deploy changes and test manually
Integration tests

This reverts commit 4f790ff.

This reverts commit 548fbf1.

jennyhliu · 2025-01-24T18:45:44Z

.eslintrc.js

@@ -62,6 +62,7 @@ module.exports = {
  },
  globals: {
    JSX: true,
+    AggregateError: false,


This is to resolve eslint error error 'AggregateError' is not defined. Later versions of eslint have resolved the error.

jennyhliu · 2025-01-27T13:15:02Z

tasks/post-to-cmr/index.js

+  const results = await pMap(
+    updatedCMRFiles,
+    (cmrFile) => publish2CMR(cmrFile, cmrSettings, cmrRevisionId),
+    { concurrency }


stopOnError: true is default, if set to false, it will change current lambda behavior

Jonathan suggested to have the promise.allsettled behavior, which is adding stopOnError: true to pMap, like
{ concurrency, stopOnError: true }, but that changes the current behavior of post-to-cmr, unit tests fail, and out of the scope of this ticket.
The comment explains why it's not added.

ohhh ok, that makes sense, thanks for lmk 🙌

jennyhliu · 2025-01-28T14:34:32Z

tasks/post-to-cmr/.nycrc.json

@@ -5,6 +5,6 @@
  ],
  "statements": 94.0,
  "functions": 88.0,
-  "branches": 97.0,
+  "branches": 88.0,


Only the handler function is not covered as before, not sure how to improve this.

…o jl/CUMULUS-3960

Nnaga1

Just some questions/comments I had, one or two small NITs, once answered + some local testing I'll approve 🙌

packages/cmrjs/src/cmr-utils.js

Nnaga1 · 2025-01-30T21:11:20Z

packages/cmrjs/src/cmr-utils.js

@@ -233,6 +243,17 @@ async function publish2CMR(cmrPublishObject, creds, cmrRevisionId) {
  throw new Error(`invalid cmrPublishObject passed to publis2CMR ${JSON.stringify(cmrPublishObject)}`);
 }

+/**


is there a reason why this wasn't here before? did we never have the case of having to remove from CMR until now or was there just another way to do it (via API or something)

mainly for my curiousity I guess

api package has _removeGranuleFromCmr which does more stuff

so this is like a mini-function of that kinda where it only deletes, sounds good and pretty useful maybe 👍

tasks/post-to-cmr/README.md

Nnaga1 · 2025-01-30T21:18:15Z

tasks/post-to-cmr/index.js

+ * @throws {Error} - Error from CMR request
+ */
+async function removeGranuleFromCmr({ granules, cmrSettings, concurrency }) {
+  const granulesToUnpublish = granules.filter((granule) => granule.published || !!granule.cmrLink);


its considered published if it doesnt have a granule.cmrLink ?

In theory published granule should have cmrLink, but we should be able to delete the granule from cmr if we don't have the link

tasks/post-to-cmr/index.js

Nnaga1 · 2025-01-30T21:27:45Z

tasks/post-to-cmr/tests/cmr-test.js

+    cmrClient.CMR.prototype.ingestGranule.restore();
+  });
+
+  await s3PutObject({


I get the other calls/functions but what does this s3PutObject do in the case of the test?

postToCMR function reads the cmr metadata file and ingest to CMR.

Nnaga1

Thanks for addressing my questions and comments! LGTM 🚀 🎈

jennyhliu added 2 commits January 21, 2025 18:34

CUMULUS-3960: Updated PostToCmr task to be able to republish granules

acd395b

upgrade localstack to fix aws-sdk error

4f790ff

jennyhliu added the work in progress label Jan 21, 2025

jennyhliu added 6 commits January 21, 2025 19:37

fix code

e9e1831

localstack return undefined for sqs.receiveMessage

548fbf1

unpublish granule without cmrLink

e356b99

Revert "upgrade localstack to fix aws-sdk error"

863fe9b

This reverts commit 4f790ff.

Revert "localstack return undefined for sqs.receiveMessage"

6050fde

This reverts commit 548fbf1.

aggregateError add configuration

c7d9b94

jennyhliu commented Jan 24, 2025

View reviewed changes

jennyhliu added 2 commits January 24, 2025 19:33

fix pass in parameter

13fa33f

stopOnError

d255ece

jennyhliu commented Jan 27, 2025

View reviewed changes

jennyhliu and others added 4 commits January 27, 2025 16:01

Merge branch 'master' into jl/CUMULUS-3960

08128fc

unit test

ef1a31e

remove integrationworkflow config add to unit test

384b3af

only one line not covered

7de83d4

jennyhliu commented Jan 28, 2025

View reviewed changes

update readme

958e18a

jennyhliu marked this pull request as ready for review January 29, 2025 15:10

Merge branch 'master' into jl/CUMULUS-3960

ad40a0f

jennyhliu added needs review and removed work in progress labels Jan 29, 2025

jennyhliu added 2 commits January 29, 2025 12:06

correct type

a9072ef

Merge branch 'jl/CUMULUS-3960' of https://github.com/nasa/cumulus int…

cf2103b

…o jl/CUMULUS-3960

Nnaga1 reviewed Jan 30, 2025

View reviewed changes

Nnaga1 added in review and removed needs review labels Jan 30, 2025

Nnaga1 approved these changes Jan 31, 2025

View reviewed changes

Nnaga1 removed the in review label Jan 31, 2025

jennyhliu merged commit 74a0648 into master Jan 31, 2025
3 checks passed

jennyhliu deleted the jl/CUMULUS-3960 branch January 31, 2025 17:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CUMULUS-3960: Updated PostToCmr task to be able to republish granules #3906

CUMULUS-3960: Updated PostToCmr task to be able to republish granules #3906

jennyhliu commented Jan 21, 2025 •

edited

Loading

jennyhliu Jan 24, 2025

jennyhliu Jan 27, 2025

jennyhliu Jan 31, 2025

Nnaga1 Jan 31, 2025

jennyhliu Jan 28, 2025 •

edited

Loading

Nnaga1 left a comment •

edited

Loading

Nnaga1 Jan 30, 2025

Nnaga1 Jan 30, 2025

jennyhliu Jan 31, 2025

Nnaga1 Jan 31, 2025

Nnaga1 Jan 30, 2025

jennyhliu Jan 31, 2025

Nnaga1 Jan 30, 2025

jennyhliu Jan 31, 2025

Nnaga1 left a comment

CUMULUS-3960: Updated PostToCmr task to be able to republish granules #3906

CUMULUS-3960: Updated PostToCmr task to be able to republish granules #3906

Conversation

jennyhliu commented Jan 21, 2025 • edited Loading

Changes

Testing

PR Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jennyhliu Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

Nnaga1 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nnaga1 left a comment

Choose a reason for hiding this comment

jennyhliu commented Jan 21, 2025 •

edited

Loading

jennyhliu Jan 28, 2025 •

edited

Loading

Nnaga1 left a comment •

edited

Loading