Releases: nasa/cumulus
Releases · nasa/cumulus
v1.9.0
Please note additional information and upgrade instructions here
Added
- CUMULUS-712 - Added integration tests verifying expected behavior in workflows
- GITC-776-2 - Add support for versioned collections
Fixed
- CUMULUS-832
- Fixed indentation in example config.yml in
@cumulus/deployment
- Fixed issue with new deployment using the default distribution endpoint in
@cumulus/deployment
and@cumulus/api
- Fixed indentation in example config.yml in
v1.8.1
Note IAM roles should be re-deployed with this release.
Added
- Cumulus-726
- Added function to
@cumulus/integration-tests
:sfnStep
includesgetStepInput
which returns the input to the schedule event of a given step function step. - Added IAM policy
@cumulus/deployment
: Lambda processing IAM role includeskinesis::PutRecord
so step function lambdas can write to kinesis streams.
- Added function to
- Cumulus Community Edition
- Added Google OAuth authentication token logic to
@cumulus/api
. Refactored token endpoint to use environment variable flagOAUTH_PROVIDER
when determining with authentication method to use. - Added API Lambda memory configuration variable
api_lambda_memory
to@cumulus/api
and@cumulus/deployment
.
- Added Google OAuth authentication token logic to
Changed
- Cumulus-726
- Changed function in
@cumulus/api
:models/rules.js#addKinesisEventSource
was modified to call todeleteKinesisEventSource
with all required parameters (rule's name, arn and type). - Changed function in
@cumulus/integration-tests
:getStepOutput
can now be used to return output of failed steps. If users of this function want the output of a failed event, they can pass a third parametereventType
as'failure'
. This function will work as always for steps which completed successfully.
- Changed function in
Removed
-
Cumulus-726
- Configuration change to
@cumulus/deployment
: Removed default auto scaling configuration for Granules and Files DynamoDB tables.
- Configuration change to
-
CUMULUS-688
- Add integration test for ExecutionStatus
- Function addition to
@cumulus/integration-tests
:api
includesgetExecutionStatus
which returns the execution status from the Cumulus API
v1.8.0
Added
-
CUMULUS-718 Adds integration test for Kinesis triggering a workflow.
-
GITC-776-3 Added more flexibility for rules. You can now edit all fields on the rule's record
We may need to update the api documentation to reflect this. -
CUMULUS-681 - Add ingest-in-place action to granules endpoint
- new applyWorkflow action at PUT /granules/{granuleid} Applying a workflow starts an execution of the provided workflow and passes the granule record as payload.
Parameter(s):- workflow - the workflow name
- new applyWorkflow action at PUT /granules/{granuleid} Applying a workflow starts an execution of the provided workflow and passes the granule record as payload.
-
CUMULUS-685 - Add parent exeuction arn to the execution which is triggered from a parent step function
Changed
- CUMULUS-768 - Integration tests get S3 provider data from shared data folder
Fixed
- CUMULUS-746 - Move granule API correctly updates record in dynamo DB and cmr xml file
- CUMULUS-766 - Populate database fileSize field from S3 if value not present in Ingest payload
v1.7.0
Please note: Upgrade Instructions
Added
- CUMULUS-491 - Add granule reconciliation API endpoints.
- CUMULUS-480 Add suport for backup and recovery:
- Add DynamoDB tables for granules, executions and pdrs
- Add ability to write all records to S3
- Add ability to download all DynamoDB records in form json files
- Add ability to upload records to DynamoDB
- Add migration scripts for copying granule, pdr and execution records from ElasticSearch to DynamoDB
- Add IAM support for batchWrite on dynamoDB
- CUMULUS-508 -
@cumulus/deployment
cloudformation template allows for lambdas and ECS clusters to have multiple AZ availability.@cumulus/deployment
also ensures docker usesdevicemapper
storage driver.
Fixed
- CUMULUS-747 - Delete granule API doesn't delete granule files in s3 and granule in elasticsearch
- update the StreamSpecification DynamoDB tables to have StreamViewType: "NEW_AND_OLD_IMAGES"
- delete granule files in s3
- CUMULUS-398 - Fix not able to filter executions bu workflow
- CUMULUS-544 - Post to CMR task has UAT URL hard-coded
- Made configurable: PostToCmr now requires CMR_ENVIRONMENT env to be set to 'SIT' or 'OPS' for those CMR environments. Default is UAT.
v1.6.0
Please note: Upgrade Instructions
Fixed
- CUMULUS-602 - Format all logs sent to Elastic Search.
- Extract cumulus log message and index it to Elastic Search.
Added
- CUMULUS-556 - add a mechanism for creating and running migration scripts on deployment.
- CUMULUS-461 Support use of metadata date and other components in
url_path
property
Changed
- CUMULUS-477 Update bucket configuration to support multiple buckets of the same type:
- Change the structure of the buckets to allow for more than one bucket of each type. The bucket structure is now:
bucket-key:
name:
type: i.e. internal, public, etc. - Change IAM and app deployment configuration to support new bucket structure
- Update tasks and workflows to support new bucket structure
- Replace instances where buckets.internal is relied upon to either use the system bucket or a configured bucket
- Move IAM template to the deployment package. NOTE: You now have to specify '--template node_modules/@cumulus/deployment/iam' in your IAM deployment
- Add IAM cloudformation template support to filter buckets by type
- Change the structure of the buckets to allow for more than one bucket of each type. The bucket structure is now:
1.5.5
Added
- CUMULUS-530 - PDR tracking through Queue-granules
- Add optional
pdr
property to the sync-granule task's input config and output payload.
- Add optional
- CUMULUS-548 - Create a Lambda task that generates EMS distribution reports
- In order to supply EMS Distribution Reports, you must enable S3 Server
Access Logging on any S3 buckets used for distribution. See How Do I Enable Server Access Logging for an S3 Bucket?
The "Target bucket" setting should point at the Cumulus internal bucket.
The "Target prefix" should be
"<STACK_NAME>/ems-distribution/s3-server-access-logs/", where "STACK_NAME"
is replaced with the name of your Cumulus stack.
- In order to supply EMS Distribution Reports, you must enable S3 Server
Fixed
- CUMULUS-546 - Kinesis Consumer should catch and log invalid JSON
- Kinesis Consumer lambda catches and logs errors so that consumer doesn't get stuck in a loop re-processing bad json records.
- EMS report filenames are now based on their start time instead of the time
instead of the time that the report was generated
1.5.4
Added
- CUMULUS-535 - EMS Ingest, Archive, Archive Delete reports
- Add lambda EmsReport to create daily EMS Ingest, Archive, Archive Delete reports
- ems.provider property added to
@cumulus/deployment/app/config.yml
.
To change the provider name, please addems: provider
property toapp/config.yml
.
1.5.3
Fixed
- CUMULUS-557 - "Add dataType to DiscoverGranules output"
- Granules discovered by the DiscoverGranules task now include dataType
- dataType is now a required property for granules used as input to the
QueueGranules task
- CUMULUS-550 Update deployment app/config.yml to force elasticsearch updates for deleted granules
1.5.2
Fixed
- CUMULUS-514 - "Unable to Delete the Granules"
- updated cmrjs.deleteConcept to return success if the record is not found
in CMR.
- updated cmrjs.deleteConcept to return success if the record is not found
Added
- CUMULUS-547 - The distribution API now includes an
"earthdataLoginUsername" query parameter when it returns a signed S3 URL - CUMULUS-527 - "parse-pdr queues up all granules and ignores regex"
- Add an optional config property to the ParsePdr task called
"granuleIdFilter". This property is a regular expression that is applied
against the filename of the first file of each granule contained in the
PDR. If the regular expression matches, then the granule is included in
the output. Defaults to '.', which will match all granules in the PDR.
- Add an optional config property to the ParsePdr task called
- File checksums in PDRs now support MD5
- Deployment support to subscribe to an SNS topic that already exists
- CUMULUS-470, CUMULUS-471 In-region S3 Policy lambda added to API to
update bucket policy for in-region access. - CUMULUS-533 Added fields to granule indexer to support EMS ingest and
archive record creation - You can now deploy cumulus without ElasticSearch. Just add
es: null
to
yourapp/config.yml
file. This is only useful for debugging purposes.
Cumulus still requires ElasticSearch to properly operate. @cumulus/integration-tests
includes and exports theaddRules
function,
which seeds rules into the DynamoDB table.- CUMULUS-534 Track deleted granules
- added
deletedgranule
type tocumulus
index. - Important Note: Force custom bootstrap to re-run by adding this to
app/config.ymles: elasticSearchMapping: 7
or use Cumulus 1.5.3 to skip this step
- added
- Added capability to support EFS in cloud formation template. Also added
optional capability to ssh to your instance and privileged lambda functions. - Added support to force discovery of PDRs that have already been processed
and filtering of selected data types @cumulus/cmrjs
uses an environment variableUSER_IP_ADDRESS
or fallback
IP address of10.0.0.0
when a public IP address is not available. This
supports lambda functions deployed into a VPC's private subnet, where no
public IP address is available.
Changed
- CUMULUS-550 Custom bootstrap automatically adds new types to index on
deployment
v1.5.1
Fixed
- add the missing dist folder to the hello-world task
- disable uglifyjs on the built version of the pdr-status-check (read: webpack-contrib/uglifyjs-webpack-plugin#264)