Stage files for v3.1.0 release (#11)

CHANGELOG.md

@@ -5,6 +5,15 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [3.1.0] - 2024-09-05
+
+### Added
+
+- Added support for retrieving reporting data from Amazon Ads API and Selling Partner API.
+- Added support for handling multiple authenticated credentials. 
+- Cataloged the reporting data in AWS Glue using the Data Lake. 
+
+
 ## [3.0.0] - 2024-05-30
 
 ### Added

NOTICE.txt

@@ -66,7 +66,8 @@ pytest-cov                                    MIT License
 pytest-env                                    MIT License                                     
 pytest-mock                                   MIT License                                     
 python-dateutil                               Apache Software License; BSD License            
-pytz                                          MIT License                                     
+pytz                                          MIT License
+pyspark                                       Apache Software License
 referencing                                   MIT License                                     
 requests                                      Apache Software License                         
 responses                                     Apache 2.0                                      

README.md

@@ -12,7 +12,13 @@ The following images show the architecture of the solution and its components
 ![Architecture 2](images/architecture-wfm.png)
 *Workflow Manager* 
 
-![Architecture 3](images/architecture-datalake.png)
+![Architecture 4](images/architecture-amazon-ads-reporting.png)
+*Amazon Ads Reporting*
+
+![Architecture 5](images/architecture-selling-partner-reporting.png)
+*Selling Partner Reporting*
+
+![Architecture 6](images/architecture-datalake.png)
 *Data Lake*
 
 **Note**: From v2.0.0, AWS CloudFormation template resources are created by the [AWS CDK](https://aws.amazon.com/cdk/)
@@ -48,9 +54,9 @@ the AWS Well-Architected Framework.
 
 ---
 
-### IAM Roles for Installation and Operation
+### IAM Roles for Installation and Admin Operation
 
-An IAM policy for installing the solution is listed within the `IAM_POLICY_INSTALL.json` file. An IAM policy for operating the solution is created on stack deployment with a name prefix of `{stack-name}-adminpolicy`. A link to this policy can be found in the Outputs window of your Cloudformation stack under the AdminPolicyOutput key. Note: the policy generated should be used as a guide. Please review it as it may need to be amended in order to fit your specific use case.
+An IAM policy for installing the solution is listed within the `IAM_POLICY_INSTALL.json` file. IAM policies for operating the solution as an admin are created on stack deployment and can be found in the Outputs window of your Cloudformation stack under the `AdminPolicyOutput` key. Note: the policies generated should be used as a guide. Please review them as they may need to be amended in order to fit your specific use case.
 
 These JSON files can be used to create a JSON policy in AWS IAM to scope the actions available to a user so they can install and operate the solution.
 

SECURITY.md

@@ -0,0 +1,7 @@
+Reporting Security Issues
+----------------------------------------------------------------------------------------------------------
+We take all security reports seriously. When we receive such reports, we will investigate and
+subsequently address any potential vulnerabilities as quickly as possible. If you discover a potential
+security issue in this project, please notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/) or
+directly via email to [AWS Security](mailto:aws-security@amazon.com). Please do not create a public GitHub issue in this project.
+

deployment/build-s3-dist.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+#
+# This assumes all of the OS-level configuration has been completed and git repo has already been cloned
+#
+# This script should be run from the repo's deployment directory
+# cd deployment
+# ./build-s3-dist.sh source-bucket-base-name solution-name version-code
+#
+# Paramenters:
+#  - source-bucket-base-name: Name for the S3 bucket location where the template will source the Lambda
+#    code from. The template will append '-[region_name]' to this bucket name.
+#    For example: ./build-s3-dist.sh solutions v1.0.0
+#    The template will then expect the source code to be located in the solutions-[region_name] bucket
+#
+#  - solution-name: name of the solution for consistency
+#
+#  - version-code: version of the package
+
+# set -euo pipefail
+
+# Check to see if input has been provided:
+if [ -z "$1" ] || [ -z "$2" ] || [ -z "$3" ]; then
+    echo "Please provide the base source bucket name, trademark approved solution name and version where the lambda code will eventually reside."
+    echo "For example: ./build-s3-dist.sh solutions trademarked-solution-name v1.0.0"
+    exit 1
+fi
+
+SOLUTION_ID="SO0193"
+
+# Get reference for all important folders
+template_dir="$PWD"
+template_dist_dir="$template_dir/global-s3-assets"
+build_dist_dir="$template_dir/regional-s3-assets"
+source_dir="$template_dir/../source"
+cdk_out="$template_dir/cdk.out"
+venv_folder=".venv-temp"
+
+echo "------------------------------------------------------------------------------"
+echo "[Init] Clean old dist, node_modules and bower_components folders"
+echo "------------------------------------------------------------------------------"
+echo "rm -rf $template_dist_dir"
+rm -rf $template_dist_dir
+echo "mkdir -p $template_dist_dir"
+mkdir -p $template_dist_dir
+echo "rm -rf $build_dist_dir"
+rm -rf $build_dist_dir
+echo "mkdir -p $build_dist_dir"
+mkdir -p $build_dist_dir
+# remove old cdk.out folder before build
+rm -rf $cdk_out
+
+echo "------------------------------------------------------------------------------"
+echo "[Rebuild] CDK Solution"
+echo "------------------------------------------------------------------------------"
+
+build_venv_name=".venv_build"
+build_venv_dir="$template_dir/$build_venv_name"
+
+# clean up testing venv if present
+rm -rf $build_venv_dir
+
+# check if we need a new testing venv
+python3 ./venv_check.py
+if [ $? == 1 ]; then
+    echo "------------------------------------------------------------------------------"
+    echo "[Env] Create clean virtual environment and install dependencies"
+    echo "------------------------------------------------------------------------------"
+    cd $root_dir
+    if [ -d $venv_folder ]; then
+      rm -rf $venv_folder
+    fi
+    python3.11 -m venv $venv_folder
+    source $venv_folder/bin/activate
+
+    using_test_venv=1
+    # configure the environment
+    cd $source_dir
+    pip install --upgrade pip
+    pip install -r $source_dir/requirements.txt
+else
+  using_test_venv=0
+  echo "------------------------------------------------------------------------------"
+  echo "[Env] Using active virtual environment for tests"
+  echo "------------------------------------------------------------------------------"
+  python_version=$(python --version 2>&1 | cut -d ' ' -f 2)
+  if [[ "$python_version" != "11"* ]]; then
+    echo "You are using Python version $python_version. Python version 11 is required."
+    echo "Update your environment or run tests again without an active environment."
+    exit 1
+  fi
+  echo ''
+fi
+
+# generate the templates (unbundled)
+cd $source_dir/infrastructure
+cdk synth -o $cdk_out

deployment/run-unit-tests.sh

@@ -54,7 +54,7 @@ else
   echo "[Env] Using active virtual environment for tests"
   echo "------------------------------------------------------------------------------"
   python_version=$(python --version 2>&1 | cut -d ' ' -f 2)
-  if [[ "$python_version" != "11"* ]]; then
+  if [[ "$python_version" != 3.11.* ]]; then
     echo "You are using Python version $python_version. Python version 11 is required."
     echo "Update your environment or run tests again without an active environment."
     exit 1

solution-manifest.yaml

@@ -1,6 +1,6 @@
 id: SO0193
 name: amazon-marketing-cloud-insights-on-aws
-version: v3.0.0
+version: v3.1.0
 cloudformation_templates:
   - template: amazon-marketing-cloud-insights.template
     main_template: true

source/cdk_solution_helper_py/helpers_cdk/aws_solutions/cdk/scripts/build_s3_cdk_dist.py

@@ -46,6 +46,9 @@ def __post_init__(self):
         self.source_dir = os.path.normpath(
             os.path.join(self.template_dir, os.pardir, "source")
         )
+        self.images_dir = os.path.normpath(
+            os.path.join(self.template_dir, os.pardir, "images")
+        )
         self.infrastructure_dir = os.path.join(self.source_dir, "infrastructure")
         self.open_source_dir = os.path.join(self.template_dir, "open-source")
         self.github_dir = os.path.normpath(
@@ -249,18 +252,20 @@ def source_code_package(ctx, ignore, solution_name):
         "CHANGELOG.md",
         ".gitignore",
         "solution-manifest.yaml",
-        "IAM_POLICY_INSTALL.json"
+        "IAM_POLICY_INSTALL.json",
+        "SECURITY.md"
     ]
 
-    # copy source directory
+    # copy required full directories
     try:
         copytree(
             env.source_dir, os.path.join(env.open_source_dir, "source"), ignore=ignored
         )
         copytree(env.github_dir, os.path.join(env.open_source_dir, ".github"))
+        copytree(env.images_dir, os.path.join(env.open_source_dir, "images"))
     except FileNotFoundError:
         raise click.ClickException(
-            "The solution requires a `source` folder and a `.github` folder"
+            "The solution requires a `source` folder, `.github` folder, and `images` folder"
         )
 
     # copy all required files
@@ -274,17 +279,19 @@ def source_code_package(ctx, ignore, solution_name):
                 f"The solution is missing the required file {name}"
             )
 
-    # copy the required run-unit-tests.sh
+    # copy the required deployment directory files
     (Path(env.open_source_dir) / "deployment").mkdir()
-    try:
-        shutil.copyfile(
-            Path(env.template_dir) / "run-unit-tests.sh",
-            Path(env.open_source_dir) / "deployment" / "run-unit-tests.sh",
-        )
-    except FileNotFoundError:
-        raise click.ClickException(
-            f"The solution is missing deployment/run-unit-tests.sh"
-        )
+    deployment_files = ["run-unit-tests.sh", "venv_check.py", "build-s3-dist.sh"]
+    for file in deployment_files:
+        try:
+            shutil.copyfile(
+                Path(env.template_dir) / file,
+                Path(env.open_source_dir) / "deployment" / file,
+            )
+        except FileNotFoundError:
+            raise click.ClickException(
+                f"The solution is missing deployment/{file}"
+            )
 
     shutil.make_archive(
         base_name=os.path.join(env.template_dir, solution_name),

source/cdk_solution_helper_py/helpers_cdk/aws_solutions/cdk/stepfunctions/solutionstep.py

@@ -119,17 +119,17 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs):
             if libraries and any(not l.exists() for l in libraries):
                 raise ValueError(f"libraries provided, but do not exist at {libraries}")
 
-            function = kwargs.pop("function")
+            _function = kwargs.pop("function")
             kwargs["layers"] = kwargs.get("layers", [])
             kwargs["tracing"] = Tracing.ACTIVE
             kwargs["timeout"] = Duration.seconds(15)
-            kwargs["runtime"] = Runtime("python3.9", RuntimeFamily.PYTHON)
+            kwargs["runtime"] = Runtime("python3.11", RuntimeFamily.PYTHON)
 
             super().__init__(
                 scope,
                 construct_id,
                 entrypoint,
-                function,
+                _function,
                 libraries=libraries,
                 **kwargs,
             )