v18.5.0

[v18.5.0] 2024-10-03

Migration Notes

CUMULUS-3536 Upgrading from Aurora Serverless V1 to V2

• The updates in CUMULUS-3536 require an upgrade of the postgres database.
  Please follow [Upgrading from Aurora Serverless V1 to V2]
  (https://nasa.github.io/cumulus/docs/next/upgrade-notes/serverless-v2-upgrade)

Added

• CUMULUS-3536
  • Added rejectUnauthorized = false to db-provision-user-database as the Lambda
    does not have the Serverless v2 SSL certifications installed.

Changed

• CUMULUS-3725
  • Updated the default parameter group for cumulus-rds-tf to set force_ssl
    to 0. This setting for the Aurora Serverless v2 database allows non-SSL
    connections to the database, and is intended to be a temporary solution
    until Cumulus has been updated to import the RDS rds-ca-rsa2048-g1 CA bundles in Lambda environments.
    See CUMULUS-3724.

Fixed

• CUMULUS-3901
  • Fix error checking in @cumulus/errors to use Error.name in addition to Error.code
• CUMULUS-3824
  • Added the missing double quote in ecs_cluster autoscaling cf template
• CUMULUS-3846
  • improve reliability of unit tests
    • tests for granules api get requests separated out to new file
    • cleanup of granule database resources to ensure no overlap
    • ensure uniqueness of execution names from getWorkflowNameIntersectFromGranuleIds
    • increase timeout in aws-client tests
• Snyk
  • Upgraded moment from 2.29.4 to 2.30.1
  • Upgraded pg from ~8.10 to ~8.12

v18.3.5

[v18.3.5] 2024-09-19

• CUMULUS-3901
  • Fix error checking in @cumulus/errors to use Error.name in addition to Error.code
  • Fix async Operation migration to include Migration Count Report

v19.0.0

Deprecated

This release has been deprecated in favor of the 18.5->19.1 release series.

[v19.0.0] 2024-08-28

Dashboard Requirement

• Cumulus Dashboard v12.2.0

Breaking Changes

• This release includes Replace ElasicSearch Phase 1 updates, we no longer save collection/granule/execution records to
  ElasticSearch, the collections/granules/executions API endpoints are updated to perform operations on the postgres database.

Migration Notes

CUMULUS-3792 Add database indexes. Please follow the instructions before upgrading Cumulus

• The updates in CUMULUS-3792 require a manual update to the postgres database in the production environment.
  Please follow [Update Table Indexes for CUMULUS-3792]
  (https://nasa.github.io/cumulus/docs/next/upgrade-notes/update_table_indexes_CUMULUS_3792)

Replace ElasticSearch Phase 1

• CUMULUS-3238
  • Removed elasticsearch dependency from collections endpoint
• CUMULUS-3239
  • Updated executions list api endpoint and added ExecutionSearch class to query postgres
• CUMULUS-3240
  • Removed Elasticsearch dependency from executions endpoints
• CUMULUS-3639
  • Updated /collections/active endpoint to query postgres
• CUMULUS-3640
  • Removed elasticsearch dependency from granules endpoint
• CUMULUS-3641
  • Updated collections api endpoint to query postgres instead of elasticsearch except if includeStats is in the query parameters
• CUMULUS-3642
  • Adjusted queries to improve performance:
    • Used count(*) over count(id) to count rows
    • Estimated row count for large tables (granules and executions) by default for basic query
  • Updated stats summary to default to the last day
  • Updated ExecutionSearch to not include asyncOperationId by default
• CUMULUS-3688
  • Updated stats api endpoint to query postgres instead of elasticsearch
• CUMULUS-3689
  • Updated stats/aggregate api endpoint to query postgres instead of elasticsearch
  • Created a new StatsSearch class for querying postgres with the stats endpoint
• CUMULUS-3692
  • Added @cumulus/db/src/search BaseSearch and GranuleSearch classes to
    support basic queries for granules
  • Updated granules List endpoint to query postgres for basic queries
• CUMULUS-3693
  • Added functionality to @cumulus/db/src/search to support range queries
• CUMULUS-3694
  • Added functionality to @cumulus/db/src/search to support term queries
  • Updated BaseSearch and GranuleSearch classes to support term queries for granules
  • Updated granules List endpoint to search postgres
• CUMULUS-3695
  • Updated granule list api endpoint and BaseSearch class to handle sort fields
• CUMULUS-3696
  • Added functionality to @cumulus/db/src/search to support terms, not and exists queries
• CUMULUS-3699
  • Updated collections api endpoint to be able to support includeStats query string parameter
• CUMULUS-3792
  • Added database indexes to improve search performance

v18.3.4

[v18.3.4] 2024-08-27

Please note changes in v18.3.4 may not yet be released in future versions, as this
is a backport/patch release on the v18.3.x series of releases. Updates that are
included in the future will have a corresponding CHANGELOG entry in future releases.

Changed

• CUMULUS-3841
  • Increased fetchRules page size to default to 100 instead of 10. This improves overall query time when fetching all rules such as in sqsMessageConsumer.

v18.4.0

[v18.4.0] 2024-08-23

Migration Notes

Important
This release requires manual intervention before applying the Terraform changes. The two tickets below require manual database updates including adding indices, modifying table constraints, and updating types. Please read and complete the steps outlined in the following two tickets, CUMULUS-3320 and CUMULUS-3449 before deploying v18.4.0.

CUMULUS-3320 Update executions table

The work for CUMULUS-3320 required index updates as well as a modification of a
table constraint. To install the update containing these changes you should:

• Pre-generate the indexes on the execution table. This can be done via manual
  procedure prior to upgrading without downtime, or done more quickly before or
  during upgrade with downtime.
• Update the executions_parent_cumulus_id_foreign constraint. This will
  require downtime as updating the constraint requires a table write lock, and
  the update may take some time.

Deployments with low volume databases and low activity and/or test/development
environments should be able to install these updates via the normal automatic
Cumulus deployment process.

Please carefully review the migration process documentation. Failure to
make these updates properly will likely result in deployment failure and/or
degraded execution table operations.

CUMULUS-3449 Please follow the instructions before upgrading Cumulus

• The updates in CUMULUS-3449 requires manual update to postgres database in
  production environment. Please follow Update Cumulus_id Type and
  Indexes

Breaking Changes

Added

• CUMULUS-3320
  • Added endpoint /executions/bulkDeleteExecutionsByCollection to allow
    bulk deletion of executions from elasticsearch by collectionId
  • Added Bulk Execution Delete migration type to async operations types
• CUMULUS-3608
  • Exposes variables for sqs_message_consumer_watcher messageLimit and timeLimit configurations. Descriptions
    of the variables here include notes on usage and what users should
    consider if configuring something other than the default values.
• CUMULUS-3449
  • Updated the following database columns to BIGINT: executions.cumulus_id, executions.parent_cumulus_id,
    files.granule_cumulus_id, granules_executions.granule_cumulus_id, granules_executions.execution_cumulus_id
    and pdrs.execution_cumulus_id
  • Changed granules table unique constraint to granules_collection_cumulus_id_granule_id_unique
  • Added indexes granules_granule_id_index and granules_provider_collection_cumulus_id_granule_id_index
    to granules table

Changed

• CUMULUS-3320
  • Updated executions table (please see Migration section and Upgrade
    Instructions for more information) to:
    • Add index on collection_cumulus_id
    • Add index on parent_cumulus_id
    • Update executions_parent_cumulus_id_foreign constraint to add ON DELETE SET NULL. This change will cause deletions in the execution table to
      allow deletion of parent executions, when this occurs the child will have
      it's parent reference set to NULL as part of the deletion operations.
• CUMULUS-3449
  • Updated @cumulus/db package and configure knex hook postProcessResponse to convert the return string
    from columns ending with "cumulus_id" to number.
• CUMULUS-3841
  • Increased fetchRules page size to default to 100 instead of 10. This improves overall query time when
    fetching all rules such as in sqsMessageConsumer.

Fixed

• CUMULUS-3817
  • updated applicable @AWS-SDK dependencies to 3.621.0 to remove inherited vulnerability from fast-xml-parser
• CUMULUS-3320
  • Execution database deletions by cumulus_id should have greatly improved
    performance as a table scan will no longer be required for each record
    deletion to validate parent-child relationships
• CUMULUS-3818
  • Fixes default value (updated to tag 52) for async-operation-image in tf-modules/cumulus.
• CUMULUS-3840
  • Fixed @cumulus/api/bin/serve to correctly use EsClient.

v18.3.3

Fixed

• CUMULUS-3824
  • Changed the ECS docker storage driver to overlay2, since devicemapper is removed in Docker Engine v25.0.
  • Removed ecs_docker_storage_driver property from cumulus module.
• CUMULUS-3836
  • Terraform configuration for cleanExecutions now correctly configures ES_HOST and lambda security group

v18.3.2

[v18.3.2] 2024-07-24

Added

• CUMULUS-3700
  • Added volume_type option to elasticsearch_config in the
    data-persistance module to allow configuration of the EBS volume type for
    Elasticsarch; default remains gp2.
• CUMULUS-3424
  • Exposed auto_pause and seconds_until_auto_pause variables in
    cumulus-rds-tf module to modify aws_rds_cluster scaling_configuration
• CUMULUS-3760
  • Added guidance for handling large backlog of es executions
• CUMULUS-3742
  • Script for dumping data into postgres database for testing and replicating issues
• CUMULUS-3385
  • Added generate_db_executions to dump large scale postgres executions

Changed

• CUMULUS-3385
  • updated cleanExecutions lambda to clean up postgres execution payloads
  • updated cleanExecutions lambda with configurable limit to control for large size
• NDCUM-1051
  • Modified addHyraxUrlToUmmG to test whether the provide Hyrax URL is already included in the metadata, and if so return the metadata unaltered.
  • Modified addHyraxUrlToEcho10 to test whether the provide Hyrax URL is already included in the metadata, and if so return the metadata unaltered.

Fixed

• CUMULUS-3807
  • Pinned @aws-sdk/client-s3 to 3.614 to address timeout/bug in s3().listObjectsV2
• CUMULUS-3787
  • Fixed developer-side bug causing some ts errors to be swallowed in CI
• CUMULUS-3785
  • Fixed SftpProviderClient not awaiting decryptBase64String with AWS KMS
  • Fixed method typo in @cumulus/api/endpoints/dashboard.js
• CUMULUS-3385
  • fixed cleanExecutions lambda to clean up elasticsearch execution payloads
• CUMULUS-3326
  • Updated update-granules-cmr-metadata-file-links task to update the file size of the update metadata file and remove the invalidated checksum associated with this file.

v18.3.1

Migration Notes

CUMULUS-3433 Update to node.js v20

The following applies only to users with a custom value configured for
async_operation_image:

• As part of the node v20 update process, a new version (52) of the Core
  async-operation container was published - cumuluss/async
  operation The
  default value for async_operation_image has been updated in the cumulus
  module, however if you are using an internal image repository such as ECR,
  please make sure to update your deployment configuration with the newly
  provided image.

  Users making use of a custom image configuration should note the base image
  for Core async operations must support node v20.x.

CUMULUS-3617 Migration of DLA messages should be performed after Cumulus is upgraded

Instructions for migrating old DLA (Dead Letter Archive) messages to new format:

• YYYY-MM-DD subfolders to organize by date
• new top level fields for simplified search and analysis
• captured error message

To invoke the Lambda and start the DLA migration, you can use the AWS Console or CLI:

aws lambda invoke --function-name $PREFIX-migrationHelperAsyncOperation \
  --payload $(echo '{"operationType": "DLA Migration"}' | base64) $OUTFILE

• PREFIX is your Cumulus deployment prefix.
• OUTFILE (optional) is the filepath where the Lambda output will be saved.

The Lambda will trigger an Async Operation and return an id such as:

{"id":"41c9fbbf-a031-4dd8-91cc-8ec2d8b5e31a","description":"Migrate Dead Letter Archive Messages",
"operationType":"DLA Migration","status":"RUNNING",
"taskArn":"arn:aws:ecs:us-east-1:AWSID:task/$PREFIX-CumulusECSCluster/123456789"}

which you can then query the Async Operations API
Endpoint for the
output or status of your request. If you want to directly observe the progress
of the migration as it runs, you can view the CloudWatch logs for your async
operations (e.g. PREFIX-AsyncOperationEcsLogs).

CUMULUS-3779 async_operations Docker image version upgrade

The async-operation Docker image has been updated to support Node v20 and aws-sdk v3. Users of the image will need
to update to at least async-operations:52.

CUMULUS-3776 cumulus-ecs-task Docker image version upgrade

The cumulus-ecs-task Docker image has been updated to support Node v20 and aws-sdk v3. Users of the image will need
to update to at least cumulus-ecs-task:2.1.0.

Breaking Changes

• CUMULUS-3618

  • Modified @cumulus/es-client/search.BaseSearch:
    • Removed static class method es in favor of new class for managing
      elasticsearch clients EsClient which allows for credential
      refresh/reset. Updated api/es-client code to
      utilize new pattern. Users making use of @cumulus/es-client should
      update their code to make use of the new EsClient create/initialize pattern.
    • Added helper method getEsClient to encapsulate logic to create/initialize
      a new EsClient.

• CUMULUS-2889

  • Removed unused CloudWatch Logs AWS SDK client. This change removes the CloudWatch Logs
    client from the @cumulus/aws-client package.

• CUMULUS-2890

  • Removed unused CloudWatch AWS SDK client. This change removes the CloudWatch client
    from the @cumulus/aws-client package.

• CUMULUS-3323

  • Updated @cumulus/db to by default set the ssl option for knex, and
    reject non-SSL connections via use of the rejectUnauthorized configuration
    flag. This causes all Cumulus database connections to require SSL (CA or
    self-signed) and reject connectivity if the database does not provide SSL.
    Users using serverless v1/cumulus-rds-tf should not be impacted by this
    change as certs are provided by default. Users using databases that do not
    provide SSL should update their database secret with the optional value
    disableSSL set to true
  • Updated cumulus-rds-tf to set rds.force_ssl to 1, forcing SSL enabled
    connections in the db_parameters configuration. Users of this module
    defining their own db_parameters should make this configuration change to allow only SSL
    connections to the RDS datastore.

• CUMULUS-2897

  • Removed unused Systems Manager AWS SDK client. This change removes the Systems Manager client
    from the @cumulus/aws-client package.

• CUMULUS-3779

  • Updates async_operations Docker image to Node v20 and bumps its cumulus dependencies to v18.3.0 to
    support aws-sdk v3 changes.

Added

• CUMULUS-3614
  • tf-modules/monitoring module now deploys Glue table for querying dead-letter-archive messages.
• CUMULUS-3616
  • Added user guide on querying dead-letter-archive messages using AWS Athena.
• CUMULUS-3433
  • Added importGot helper method to import got as an ESM module in
    CommmonJS typescript/webpack clients.
• CUMULUS-3606
  • Updated with additional documentation covering tunneling configuration
    using a PKCS11 provider

Changed

• CUMULUS-3735
  • Remove unused getGranuleIdsForPayload from @cumulus/api/lib
• CUMULUS-3746
  • cicd unit test error log changed to environment unique name
• CUMULUS-3717
  • Update @cumulus/ingest/HttpProviderClient to use direct injection test mocks, and remove rewire from unit tests
• CUMULUS-3720
  • add cicd unit test error logging to s3 for testing improvements
• CUMULUS-3433
  • Updated all node.js lambda dependencies to node 20.x/20.12.2
  • Modified @cumulus/ingest unit test HTTPs server to accept localhost POST
    requests, and removed nock dependency from tests involving fs.Readstream
    and got due to a likely incompatibility with changes in node v18, got,
    fs.Readstream and nock when used in combination in units
    (sindresorhus/got#2341)
  • Updated got dependency in @cumulus/ingest to use @cumulus/common
    dynamic import helper / got > v10 in CommonJS.
  • Updated all Core lambdas to use cumulus-message-adapter-js v2.2.0
• CUMULUS-3629
  • dla guarantees de-nested SQS message bodies, preferring outermost metadata as found.
  • dla uses execution Name as filename and ensures no ':' or '/' characters in name
• CUMULUS-3570
  • Updated Kinesis docs to support latest AWS UI and recommend server-side encryption.
• CUMULUS-3519
  • Updates SQS and SNS code to AWS SDK V3 Syntax
• CUMULUS-3609
  • Adds dla-migration lambda to async-operations to be used for updating existing DLA records
  • Moved hoistCumulusMessageDetails function from write-db-dlq-records-to-s3 lambda to @cumulus/message/DeadLetterMessage
• CUMULUS-3613
  • Updated writeDbRecordsDLQtoS3 lambda to write messages to YYYY-MM-DD subfolder of S3 dead letter archive.
• CUMULUS-3518
  • Update existing usage of @cumulus/aws-client lambda service to use AWS SDK v3 send syntax
  • Update Discover Granules lambda default memory to 1024 MB
• CUMULUS-3600
  • Update docs to clarify CloudFront HTTPS DIT requirements.
• CUMULUS-2892
  • Updates aws-client's EC2 client to use AWS SDK v3.
• CUMULUS-2896
  • Updated Secrets Manager code to AWS SDK v3.
• CUMULUS-2901
  • Updated STS code to AWS SDK v3.
• CUMULUS-2898
  • Update Step Functions code to AWS SDK v3
• CUMULUS-2902
  • Removes aws-sdk from es-client package by replacing credential fetching with
    the @aws-sdk/credential-providers AWS SDK v3 package.
  • Removes aws-sdk from all cumulus packages and replaces usages with AWS SDK v3 clients.
• CUMULUS-3456
  • Added stateMachineArn, executionArn, collectionId, providerId, granules, status, time, and error fields to Dead Letter Archive message
  • Added cumulusError field to records in sfEventSqsToDbRecordsDeadLetterQueue
• CUMULUS-3323
  • Added disableSSL as a valid database secret key - setting this in your database credentials will
    disable SSL for all Core database connection attempts.
  • Added rejectUnauthorized as a valid database secret key - setting
    this to false in your database credentials will allow self-signed certs/certs with an unrecognized authority.
  • Updated the default parameter group for cumulus-rds-tf to set force_ssl
    to 1. This setting for the Aurora Serverless v1 database disallows non-SSL
    connections to the database, and is intended to help enforce security
    compliance rules. This update can be opted-out by supplying a non-default
    db_parameters set in the terraform configuration.
• CUMULUS-3425
  • Update @cumulus/lzards-backup task to either respect the lzards_provider
    terraform configuration value or utilize lzardsProvider as part of the task
    workflow configuration
  • Minor refactor of @cumulus/lzards-api-client to:
    • Use proper ECMAScript import for @cumulus/launchpad-auth
    • Update incorrect docstring
• CUMULUS-3497
  • Updated example/cumulus-tf/orca.tf to use v9.0.4
• CUMULUS-3610
  • Updated aws-client's ES client to use AWS SDK v3.
• CUMULUS-3617
  • Added lambdas to migrate DLA messages to YYYY-MM-DD subfolder
  • Updated @cumulus/aws-client/S3/recursivelyDeleteS3Bucket to handle bucket with more than 1000 objects.
• *...

v18.2.2

[v18.2.2] 2024-06-4

Migration Notes

CUMULUS-3591 - SNS topics set to use encrypted storage

As part of the requirements for this ticket Cumulus Core created SNS topics are
being updated to use server-side encryption with an AWS managed key. No user
action is required, this note is being added to increase visibility re: this
modification.

Changed

• CUMULUS-3591
  • Enable server-side encryption for all SNS topcis deployed by Cumulus Core
  • Update all integration/unit tests to use encrypted SNS topics

Fixed

• CUMULUS-3547
  • Updated ECS Cluster /dev/xvdcz EBS volumes so they're encrypted.
• CUMULUS-3527
  • Added suppport for additional kex algorithms in the sftp-client.
• CUMULUS-3587
  • Ported https://github.com/scottcorgan/express-boom into API/lib to allow
    updates of sub-dependencies and maintain without refactoring errors in
    API/etc wholesale
  • Addresses CVE-2020-36604
• CUMULUS-3673
  • Fixes Granules API so that paths containing a granule and/or collection ID properly URI encode the ID.
• Audit Issues
  • Addressed CVE-2023-45133 by
    updating babel packages and .babelrc

v16.1.5

Release v16.1.5

[v16.1.5] 2024-05-09 [BACKPORT]

Please note changes in 16.1.5 may not yet be released in future versions, as this
is a backport/patch release on the 16.x series of releases. Updates that are
included in the future will have a corresponding CHANGELOG entry in future releases.

Fixed

CUMULUS-3721

• Update lambda:GetFunctionConfiguration policy statement to fix error related to resource naming.
  CUMULUS-3527

• Added support for additional kex algorithms in the sftp-client.