Merge pull request #3048 from nasa/release-11.1.4

Release v11.1.4

CHANGELOG.md

@@ -6,6 +6,122 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ## Unreleased
 
+## [v11.1.4] 2022-07-18
+
+**Please note** changes in 11.1.4 may not yet be released in future versions, as
+this is a backport and patch release on the 11.1.x series of releases. Updates that
+are included in the future will have a corresponding CHANGELOG entry in future
+releases.
+
+### MIGRATION notes
+
+
+- The changes introduced in CUMULUS-2962 will re-introduce a
+  `files_granules_cumulus_id_index` on the `files` table in the RDS database.
+  This index will be automatically created as part of the bootstrap lambda
+  function *on deployment* of the `data-persistence` module.
+
+  *In cases where the index is already applied, this update will have no effect*.
+
+  **Please Note**: In some cases where ingest is occurring at high volume levels and/or the
+  files table has > 150M file records, the migration may
+  fail on deployment due to timing required to both acquire the table state needed for the
+  migration and time to create the index given the resources available.
+
+  For reference a rx.5 large Aurora/RDS database
+  with *no activity* took roughly 6 minutes to create the index for a file table with 300M records and no active ingest, however timed out when the same migration was attempted
+  in production with possible activity on the table.
+
+  If you believe you are subject to the above consideration, you may opt to
+  manually create the `files` table index *prior* to deploying this version of
+  Core with the following procedure:
+
+  -----
+
+  - Verify you do not have the index:
+
+  ```text
+  select * from pg_indexes where tablename = 'files';
+
+   schemaname | tablename |        indexname        | tablespace |                                       indexdef
+  ------------+-----------+-------------------------+------------+---------------------------------------------------------------------------------------
+   public     | files     | files_pkey              |            | CREATE UNIQUE INDEX files_pkey ON public.files USING btree (cumulus_id)
+   public     | files     | files_bucket_key_unique |            | CREATE UNIQUE INDEX files_bucket_key_unique ON public.files USING btree (bucket, key)
+  ```
+
+  In this instance you should not see an `indexname` row with
+  `files_granules_cumulus_id_index` as the value.     If you *do*, you should be
+  clear to proceed with the installation.
+  - Quiesce ingest
+
+  Stop all ingest operations in Cumulus Core according to your operational
+  procedures.    You should validate that it appears there are no active queries that
+  appear to be inserting granules/files into the database as a secondary method
+  of evaluating the database system state:
+
+  ```text
+  select pid, query, state, wait_event_type, wait_event from pg_stat_activity where state = 'active';
+  ```
+
+  If query rows are returned with a `query` value that involves the files table,
+  make sure ingest is halted and no other granule-update activity is running on
+  the system.
+
+  Note: In rare instances if there are hung queries that are unable to resolve, it may be necessary to
+  manually use psql [Server Signaling
+  Functions](https://www.postgresql.org/docs/10/functions-admin.html#FUNCTIONS-ADMIN-SIGNAL)
+  `pg_cancel_backend` and/or
+  `pg_terminate_backend` if the migration will not complete in the next step.
+
+  - Create the Index
+
+  Run the following query to create the index.    Depending on the situation
+  this may take many minutes to complete, and you will note your CPU load and
+  disk I/O rates increase on your cluster:
+
+  ```text
+  CREATE INDEX files_granule_cumulus_id_index ON files (granule_cumulus_id);
+  ```
+
+  You should see a response like:
+
+  ```text
+  CREATE INDEX
+  ```
+
+  and can verify the index `files_granule_cumulus_id_index` was created:
+
+  ```text
+  => select * from pg_indexes where tablename = 'files';
+  schemaname | tablename |           indexname            | tablespace |                                           indexdef
+   ------------+-----------+--------------------------------+------------+----------------------------------------------------------------------------------------------
+   public     | files     | files_pkey                     |            | CREATE UNIQUE INDEX files_pkey ON public.files USING btree (cumulus_id)
+   public     | files     | files_bucket_key_unique        |            | CREATE UNIQUE INDEX files_bucket_key_unique ON public.files USING btree (bucket, key)
+   public     | files     | files_granule_cumulus_id_index |            | CREATE INDEX files_granule_cumulus_id_index ON public.files USING btree (granule_cumulus_id)
+  (3 rows)
+  ```
+
+  - Once this is complete, you may deploy this version of Cumulus as you
+    normally would.
+  **If you are unable to stop ingest for the above procedure** *and* cannot
+  migrate with deployment, you may be able to manually create the index while
+  writes are ongoing using postgres's `CONCURRENTLY` option for `CREATE INDEX`.
+  This can have significant impacts on CPU/write IO, particularly if you are
+  already using a significant amount of your cluster resources, and may result
+  in failed writes or an unexpected index/database state.
+
+  PostgreSQL's
+  [documentation](https://www.postgresql.org/docs/10/sql-createindex.html#SQL-CREATEINDEX-CONCURRENTLY)
+  provides more information on this option.   Please be aware it is
+  **unsupported** by Cumulus at this time, so community members that opt to go
+  this route should proceed with caution.
+
+  -----
+
+### Changed
+
+- Updated Moment.js package to 2.29.4 to address security vulnerability
+
 ## [v11.1.3] 2022-06-17
 
 **Please note** changes in 11.1.3 may not yet be released in future versions, as
@@ -15,6 +131,8 @@ releases.
 
 ### Notable changes
 
+- **CUMULUS-2962**
+  - Re-added database structural migration to `files` table to add an index on `granule_cumulus_id`
 - **CUMULUS-2929**
   - Updated `move-granule` task to check the optional collection configuration parameter
     `meta.granuleMetadataFileExtension` to determine the granule metadata file.
@@ -5956,7 +6074,8 @@ Note: There was an issue publishing 1.12.0. Upgrade to 1.12.1.
 
 ## [v1.0.0] - 2018-02-23
 
-[unreleased]: https://github.com/nasa/cumulus/compare/v11.1.3...HEAD
+[unreleased]: https://github.com/nasa/cumulus/compare/v11.1.4...HEAD
+[v11.1.4]: https://github.com/nasa/cumulus/compare/v11.1.3...v11.1.4
 [v11.1.3]: https://github.com/nasa/cumulus/compare/v11.1.2...v11.1.3
 [v11.1.2]: https://github.com/nasa/cumulus/compare/v11.1.1...v11.1.2
 [v11.1.1]: https://github.com/nasa/cumulus/compare/v11.1.0...v11.1.1

example/lambdas/asyncOperations/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-async-operations",
-  "version": "11.1.3",
+  "version": "11.1.4",
   "description": "AsyncOperations Test Lambda",
   "main": "index.js",
   "private": true,

example/lambdas/ftpPopulateTestLambda/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/ftp-populate-test-lambda",
-  "version": "11.1.3",
+  "version": "11.1.4",
   "description": "FTP Population Utility Lambda",
   "main": "index.js",
   "private": true,
@@ -19,12 +19,12 @@
     "access": "private"
   },
   "dependencies": {
-    "@cumulus/api": "11.1.3",
-    "@cumulus/api-client": "11.1.3",
-    "@cumulus/common": "11.1.3",
-    "@cumulus/integration-tests": "11.1.3",
-    "@cumulus/logger": "11.1.3",
-    "@cumulus/test-data": "11.1.3",
+    "@cumulus/api": "11.1.4",
+    "@cumulus/api-client": "11.1.4",
+    "@cumulus/common": "11.1.4",
+    "@cumulus/integration-tests": "11.1.4",
+    "@cumulus/logger": "11.1.4",
+    "@cumulus/test-data": "11.1.4",
     "aws-sdk": "^2.585.0",
     "fs-extra": "^9.0.0",
     "js-yaml": "^3.13.1",

example/lambdas/python-processing/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/python-process-activity",
   "private": true,
-  "version": "11.1.3",
+  "version": "11.1.4",
   "description": "Python reference activity",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/lambdas/python-reference-activity",
   "repository": {

example/lambdas/python-reference-activity/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/python-reference-activity",
   "private": true,
-  "version": "11.1.3",
+  "version": "11.1.4",
   "description": "Python reference activity",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/lambdas/python-reference-activity",
   "repository": {

example/lambdas/python-reference-task/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/python-reference-task",
   "private": true,
-  "version": "11.1.3",
+  "version": "11.1.4",
   "description": "Python reference task",
   "main": "index.js",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/lambdas/python-reference-task",

example/lambdas/s3AccessTest/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-s3-access",
-  "version": "11.1.3",
+  "version": "11.1.4",
   "description": "S3 Access Test Lambda",
   "main": "index.js",
   "private": true,

example/lambdas/snsS3Test/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-sns-s3",
-  "version": "11.1.3",
+  "version": "11.1.4",
   "description": "SNS to S3 Test Lambda",
   "main": "index.js",
   "private": true,

example/lambdas/versionUpTest/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/test-version-up",
-  "version": "11.1.3",
+  "version": "11.1.4",
   "description": "Version Up Test Lambda",
   "main": "index.js",
   "private": true,

example/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/cumulus-integration-tests",
-  "version": "11.1.3",
+  "version": "11.1.4",
   "description": "Cumulus Integration Test Deployment",
   "private": true,
   "main": "index.js",
@@ -45,32 +45,32 @@
     ]
   },
   "dependencies": {
-    "@cumulus/api": "11.1.3",
-    "@cumulus/api-client": "11.1.3",
-    "@cumulus/async-operations": "11.1.3",
-    "@cumulus/aws-client": "11.1.3",
-    "@cumulus/checksum": "11.1.3",
-    "@cumulus/cmr-client": "11.1.3",
-    "@cumulus/cmrjs": "11.1.3",
-    "@cumulus/common": "11.1.3",
-    "@cumulus/discover-granules": "11.1.3",
-    "@cumulus/discover-pdrs": "11.1.3",
-    "@cumulus/files-to-granules": "11.1.3",
-    "@cumulus/hello-world": "11.1.3",
-    "@cumulus/ingest": "11.1.3",
-    "@cumulus/integration-tests": "11.1.3",
-    "@cumulus/message": "11.1.3",
-    "@cumulus/move-granules": "11.1.3",
-    "@cumulus/parse-pdr": "11.1.3",
-    "@cumulus/pdr-status-check": "11.1.3",
-    "@cumulus/post-to-cmr": "11.1.3",
-    "@cumulus/queue-granules": "11.1.3",
-    "@cumulus/queue-pdrs": "11.1.3",
-    "@cumulus/sf-sqs-report": "11.1.3",
-    "@cumulus/sync-granule": "11.1.3",
-    "@cumulus/test-processing": "11.1.3"
+    "@cumulus/api": "11.1.4",
+    "@cumulus/api-client": "11.1.4",
+    "@cumulus/async-operations": "11.1.4",
+    "@cumulus/aws-client": "11.1.4",
+    "@cumulus/checksum": "11.1.4",
+    "@cumulus/cmr-client": "11.1.4",
+    "@cumulus/cmrjs": "11.1.4",
+    "@cumulus/common": "11.1.4",
+    "@cumulus/discover-granules": "11.1.4",
+    "@cumulus/discover-pdrs": "11.1.4",
+    "@cumulus/files-to-granules": "11.1.4",
+    "@cumulus/hello-world": "11.1.4",
+    "@cumulus/ingest": "11.1.4",
+    "@cumulus/integration-tests": "11.1.4",
+    "@cumulus/message": "11.1.4",
+    "@cumulus/move-granules": "11.1.4",
+    "@cumulus/parse-pdr": "11.1.4",
+    "@cumulus/pdr-status-check": "11.1.4",
+    "@cumulus/post-to-cmr": "11.1.4",
+    "@cumulus/queue-granules": "11.1.4",
+    "@cumulus/queue-pdrs": "11.1.4",
+    "@cumulus/sf-sqs-report": "11.1.4",
+    "@cumulus/sync-granule": "11.1.4",
+    "@cumulus/test-processing": "11.1.4"
   },
   "devDependencies": {
-    "@cumulus/test-data": "11.1.3"
+    "@cumulus/test-data": "11.1.4"
   }
 }

example/scripts/generate_ingest/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/generate_ingest",
   "private": true,
-  "version": "11.1.3",
+  "version": "11.1.4",
   "description": "Script to generate test data for scaled ingest",
   "keywords": [
     "GIBS",
@@ -22,8 +22,8 @@
     "directory": "packages/types"
   },
   "dependencies": {
-    "@cumulus/aws-client": "11.1.3",
-    "@cumulus/common": "11.1.3"
+    "@cumulus/aws-client": "11.1.4",
+    "@cumulus/common": "11.1.4"
   },
   "author": "Cumulus Authors",
   "license": "Apache-2.0"

example/scripts/lib/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@cumulus/example-lib",
   "private": true,
-  "version": "11.1.3",
+  "version": "11.1.4",
   "description": "example project libs",
   "homepage": "https://github.com/nasa/cumulus/tree/master/example/scripts/lib",
   "engines": {

lambdas/data-migration1/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/data-migration1",
-  "version": "11.1.3",
+  "version": "11.1.4",
   "description": "A Lambda function used for doing data migrations",
   "license": "Apache-2.0",
   "engines": {
@@ -25,18 +25,18 @@
     "timeout": "15m"
   },
   "dependencies": {
-    "@cumulus/api": "11.1.3",
-    "@cumulus/aws-client": "11.1.3",
-    "@cumulus/common": "11.1.3",
-    "@cumulus/db": "11.1.3",
-    "@cumulus/errors": "11.1.3",
-    "@cumulus/logger": "11.1.3",
-    "@cumulus/types": "11.1.3",
+    "@cumulus/api": "11.1.4",
+    "@cumulus/aws-client": "11.1.4",
+    "@cumulus/common": "11.1.4",
+    "@cumulus/db": "11.1.4",
+    "@cumulus/errors": "11.1.4",
+    "@cumulus/logger": "11.1.4",
+    "@cumulus/types": "11.1.4",
     "knex": "0.95.15",
     "lodash": "^4.17.21",
     "pg": "^8.3.0"
   },
   "devDependencies": {
-    "@cumulus/test-data": "11.1.3"
+    "@cumulus/test-data": "11.1.4"
   }
 }

lambdas/data-migration2/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@cumulus/data-migration2",
-  "version": "11.1.3",
+  "version": "11.1.4",
   "description": "A Lambda function used for doing data migrations intended to be executed after data-migration1.",
   "license": "Apache-2.0",
   "engines": {
@@ -28,21 +28,21 @@
     "@aws-sdk/client-dynamodb": "^3.53.0",
     "@aws-sdk/lib-dynamodb": "^3.53.0",
     "@aws-sdk/util-dynamodb": "^3.53.0",
-    "@cumulus/api": "11.1.3",
-    "@cumulus/aws-client": "11.1.3",
-    "@cumulus/common": "11.1.3",
-    "@cumulus/db": "11.1.3",
-    "@cumulus/errors": "11.1.3",
-    "@cumulus/logger": "11.1.3",
-    "@cumulus/message": "11.1.3",
-    "@cumulus/types": "11.1.3",
+    "@cumulus/api": "11.1.4",
+    "@cumulus/aws-client": "11.1.4",
+    "@cumulus/common": "11.1.4",
+    "@cumulus/db": "11.1.4",
+    "@cumulus/errors": "11.1.4",
+    "@cumulus/logger": "11.1.4",
+    "@cumulus/message": "11.1.4",
+    "@cumulus/types": "11.1.4",
     "JSONStream": "1.3.5",
     "knex": "0.95.15",
     "lodash": "^4.17.21",
     "p-map": "^4.0.0",
     "pg": "^8.3.0"
   },
   "devDependencies": {
-    "@cumulus/test-data": "11.1.3"
+    "@cumulus/test-data": "11.1.4"
   }
 }