Merge pull request #161 from cfs-energy-internal/3.1.6

IDEA Release 3.1.6

CHANGELOG.md

@@ -4,6 +4,58 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [3.1.6] - 2023-10-20
+
+### Features
+
+* Create all EFA interfaces for instances that support multiple-EFA interfaces (when an HPC job sets `enable_efa` to `True`)
+* Added support for `Red Hat Enterprise Linux 9.2` and `Rocky Linux 9.2` as supported operating systems for VDI and compute nodes.
+  * **NOTE:** Subscription to `Rocky Linux 9` in AWS Marketplace (https://aws.amazon.com/marketplace/pp/prodview-ygp66mwgbl2ii) is required to access Rocky Linux AMIs.
+  * `Red Hat Enterprise Linux 9.2` and `Rocky Linux 9.2` are not currently supported as installation / infrastructure nodes.
+
+
+### Changes
+* Improvements to Active Directory join process for Windows eVDI sessions.
+  * Improved debug logging for troubleshooting
+  * Add minimum floor of `3` seconds to wait before attempting AD join
+  * Auto-retry eVDI Active Directory process to reduce the chance of a join failure in busy environments
+  * Active Directory join timers are now controlled with new configuration options
+    * `directoryservice.ad_automation.ad_join_min_sleep` - The minimum sleep interval before attempting join (per loop)
+    * `directoryservice.ad_automation.ad_join_max_sleep` - The maximum sleep interval before attempting join (per loop)
+    * `directoryservice.ad_automation.ad_join_retry_count` - The retry count (loop count) to attempt Active Directory joins
+  * Allow the `cluster-manager` to supply an `OU` and `hostname` back to the joining client.
+    * This can be used to move `hostname` generation logic and `OU` sorting to `cluster-manager` (customizations required)
+    * Auto-rename the host on AD join when `cluster-manager` supplied hostname differs from the hostname
+* Dashes (`-`) are now allowed in IDEA usernames
+  * Dashes are not permitted at the start or end of a username, or appearing consecutively
+* `cluster-manager` will now mask the content of the API request `FileBrowser.SaveFile` (used when editing/saving a file in the browser)
+* NICE DCV historic session information from the internal API is available for up to `1-hour` after the DCV server has been removed
+* NICE DCV servers can be purged from API responses after being unreachable for `900` seconds
+* Update NVIDIA GPU drivers used during installation
+  * LTSB from `470.141.03` to `470.199.02`
+  * Production `525.105.17` to `535.104.05`
+* Update OpenPBS Scheduler from `v22.05.11` to `v23.06.06`
+* Update Python from `3.9.16` to `3.9.18`
+* Update AWS CDK from `2.93.0` to `2.95.1`
+* Updated boto3 from `1.28.12` to `1.28.44`
+* Update all Classes to be compatible with `Pydantic v2`
+* Update OpenAPI spec to `3.1.0`
+* Update swagger links to newer OpenAPI (`3.1.0`) compatible versions
+* Allow `VDC Controller` IAM permissions for Auto Scaling Groups
+* Misc 3rd party package updates
+
+### Bug Fixes
+* In the File browser right-click / context menu - the `Open Selection` action didn't open the file.
+* Updated installer policies for missing IAM permissions
+* AWS tag `JobOwner` was not being set for eVDI sessions. This has been fixed.
+* Misc PEP cleanups
+
+
+### Known Caveats
+* FSxL is not supported on Rocky9 and RHEL9
+* EFA is not supported on RHEL9
+* RHEL9 / Rocky9 is not able to be used as infrastructure nodes (IDEA/NICE DCV application nodes)
+
 
 ## [3.1.5] - 2023-09-06
 
@@ -38,7 +90,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 * Restore ability to have spaces in the Project `title`
 * During `GovCloud` installation - display a list of AWS profiles for the commercial profile versus requiring the user to type it in.
 * Correct a defect that was not allowing the selection of Tenancy Choices for a Software stack.
-* Improve eVDI subnet retry logic for both Capacity exceptions abd Unsupported Instance exceptions
+* Improve eVDI subnet retry logic for both Capacity exceptions and Unsupported Instance exceptions
 * OpenSearch domains that were not completely deployed were being listed during `idea-admin.sh config generate --existing-resources`
 * When generating a stack from a session - make sure to copy the minimum storage and projects from the session. This should restore the ability to create stacks from existing sessions.
 * Fixed WebUI modal for Session Sharing permissions appearing with a dark blue header no matter what the selected theme is

IDEA_VERSION.txt

@@ -1 +1 @@
-3.1.5
+3.1.6

deployment/ecr/idea-administrator/Dockerfile

@@ -1,4 +1,4 @@
-FROM public.ecr.aws/docker/library/python:3.9.16-slim
+FROM public.ecr.aws/docker/library/python:3.9.18-slim
 
 WORKDIR /root
 
@@ -26,10 +26,15 @@ RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2
     ./aws/install && \
     rm -rf ./aws awscliv2.zip
 
-# install nvm and node
-RUN curl -sL https://deb.nodesource.com/setup_16.x | bash - \
-    && apt-get install -y nodejs \
-    && apt-get clean all
+# install node and npm
+RUN NODE_MAJOR=16 && rm -f /etc/apt/trusted.gpg.d/nodesource.gpg /etc/apt/sources.list.d/nodesource.list \
+    && apt-get install -y ca-certificates curl gnupg \
+    && curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/trusted.gpg.d/nodesource.gpg \
+    && echo "deb [signed-by=/etc/apt/trusted.gpg.d/nodesource.gpg] https://deb.nodesource.com/node_${NODE_MAJOR}.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list \
+    && echo "Package: nodejs" >> /etc/apt/preferences.d/preferences \
+    && echo "Pin: origin deb.nodesource.com" >> /etc/apt/preferences.d/preferences \
+    && echo "Pin-Priority: 1001" >> /etc/apt/preferences.d/preferences\
+    && apt update && apt-get install -y nodejs
 
 # add all packaged artifacts to container
 ARG PUBLIC_ECR_TAG

idea-admin-windows.ps1

@@ -38,7 +38,7 @@ function Verify-Command($type,$message,$command) {
 $IDEADevMode = if ($Env:IDEA_DEV_MODE) {$Env:IDEA_DEV_MODE} else {""}
 $VirtualEnv = if ($Env:VIRTUAL_ENV) {$Env:VIRTUAL_ENV} else {""}
 $ScriptDir = $PSScriptRoot
-$IDEARevision = if ($Env:IDEA_REVISION) {$Env:IDEA_REVISION} else {"v3.1.5"}
+$IDEARevision = if ($Env:IDEA_REVISION) {$Env:IDEA_REVISION} else {"v3.1.6"}
 $IDEADockerRepo = "public.ecr.aws/h5i3y8y1"
 $DocumentationError = "https://ide-on-aws.com"
 $AWSProfile = if ($Env:AWS_PROFILE) {$Env:AWS_PROFILE} else {"default"}

idea-admin.sh

@@ -28,7 +28,7 @@
 # * IDEA_DEV_MODE - Set to "true" if you are working with IDEA sources
 
 SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
-IDEA_REVISION=${IDEA_REVISION:-"v3.1.5"}
+IDEA_REVISION=${IDEA_REVISION:-"v3.1.6"}
 IDEA_DOCKER_REPO=${IDEA_DOCKER_REPO:-"public.ecr.aws/h5i3y8y1/idea-administrator"}
 IDEA_ECR_CREDS_RESET=${IDEA_ECR_CREDS_RESET:-"true"}
 IDEA_ADMIN_AWS_CREDENTIAL_PROVIDER=${IDEA_ADMIN_AWS_CREDENTIAL_PROVIDER:=""}

requirements/dev.in

@@ -8,6 +8,6 @@
 pip-tools
 invoke
 pyfiglet
-openapi-schema-pydantic
+openapi-pydantic
 pylint
 wheel~=0.38.1

requirements/dev.txt

@@ -1,16 +1,17 @@
 aiofiles==23.1.0
 alembic==1.11.1
+annotated-types==0.5.0
 arrow==1.2.3
 astroid==2.15.6
 attrs==23.1.0
 aws-cdk-asset-awscli-v1==2.2.200
 aws-cdk-asset-kubectl-v20==2.1.2
 aws-cdk-asset-node-proxy-agent-v6==2.0.1
-aws-cdk-lib==2.93.0
+aws-cdk-lib==2.95.1
 banal==1.0.6
 blinker==1.6.2
-boto3==1.28.12
-botocore==1.31.12
+boto3==1.28.44
+botocore==1.31.44
 build==0.10.0
 cacheout==0.14.1
 cachetools==5.3.1
@@ -25,7 +26,7 @@ colorama==0.4.6
 colored==2.2.3
 constructs==10.2.69
 coverage[toml]==7.2.7
-cryptography==41.0.3
+cryptography==41.0.4
 dataset==1.6.2
 decorator==5.1.1
 dill==0.3.7
@@ -61,7 +62,7 @@ mkdocs-material-extensions==1.1.1
 multidict==6.0.4
 mypy==1.4.1
 mypy-extensions==1.0.0
-openapi-schema-pydantic==1.2.4
+openapi-pydantic==0.3.2
 opensearch-py==2.2.0
 orjson==3.9.2
 packaging==23.1
@@ -77,7 +78,8 @@ publication==0.0.3
 pyasn1==0.5.0
 pyasn1-modules==0.3.0
 pycparser==2.21
-pydantic==1.10.12
+pydantic==2.3.0
+pydantic-core==2.6.3
 pyfiglet==0.8.post1
 pygments==2.15.1
 pyhocon==0.3.60
@@ -114,12 +116,12 @@ supervisor==4.2.5
 tomli==2.0.1
 tomlkit==0.12.0
 tracerite==1.1.0
-troposphere==4.3.2
+troposphere==4.4.1
 typeguard==2.13.3
 typing-extensions==4.7.1
 tzlocal==5.0.1
 ujson==5.8.0
-urllib3==1.26.16
+urllib3==1.26.18
 uvloop==0.17.0
 validators==0.20.0
 watchdog==3.0.0

requirements/idea-administrator.txt

@@ -1,15 +1,16 @@
 aiofiles==23.1.0
 alembic==1.11.1
+annotated-types==0.5.0
 arrow==1.2.3
 attrs==23.1.0
 aws-cdk-asset-awscli-v1==2.2.200
 aws-cdk-asset-kubectl-v20==2.1.2
 aws-cdk-asset-node-proxy-agent-v6==2.0.1
-aws-cdk-lib==2.93.0
+aws-cdk-lib==2.95.1
 banal==1.0.6
 blinker==1.6.2
-boto3==1.28.12
-botocore==1.31.12
+boto3==1.28.44
+botocore==1.31.44
 cacheout==0.14.1
 cattrs==23.1.2
 cdk-nag==2.27.82
@@ -20,7 +21,7 @@ charset-normalizer==3.2.0
 click==8.1.6
 colored==2.2.3
 constructs==10.2.69
-cryptography==41.0.3
+cryptography==41.0.4
 dataset==1.6.2
 decorator==5.1.1
 exceptiongroup==1.1.2
@@ -49,7 +50,8 @@ prompt-toolkit==3.0.39
 psutil==5.9.5
 publication==0.0.3
 pycparser==2.21
-pydantic==1.10.12
+pydantic==2.3.0
+pydantic-core==2.6.3
 pygments==2.15.1
 pyhocon==0.3.60
 pyjwt==2.8.0
@@ -74,12 +76,12 @@ six==1.16.0
 sqlalchemy==1.4.49
 tomli==2.0.1
 tracerite==1.1.0
-troposphere==4.3.2
+troposphere==4.4.1
 typeguard==2.13.3
 typing-extensions==4.7.1
 tzlocal==5.0.1
 ujson==5.8.0
-urllib3==1.26.16
+urllib3==1.26.18
 uvloop==0.17.0
 validators==0.20.0
 wcwidth==0.2.6

requirements/idea-cluster-manager.txt

@@ -1,17 +1,18 @@
 aiofiles==23.1.0
 alembic==1.11.1
+annotated-types==0.5.0
 arrow==1.2.3
 banal==1.0.6
 blinker==1.6.2
-boto3==1.28.12
-botocore==1.31.12
+boto3==1.28.44
+botocore==1.31.44
 cacheout==0.14.1
 certifi==2023.7.22
 cffi==1.15.1
 cfn-flip==1.3.0
 charset-normalizer==3.2.0
 click==8.1.6
-cryptography==41.0.3
+cryptography==41.0.4
 dataset==1.6.2
 decorator==5.1.1
 fastcounter==1.1.0
@@ -38,7 +39,8 @@ psutil==5.9.5
 pyasn1==0.5.0
 pyasn1-modules==0.3.0
 pycparser==2.21
-pydantic==1.10.12
+pydantic==2.3.0
+pydantic-core==2.6.3
 pygments==2.15.1
 pyhocon==0.3.60
 pyjwt==2.8.0
@@ -65,11 +67,11 @@ sqlalchemy==1.4.49
 supervisor==4.2.5
 tomli==2.0.1
 tracerite==1.1.0
-troposphere==4.3.2
+troposphere==4.4.1
 typing-extensions==4.7.1
 tzlocal==5.0.1
 ujson==5.8.0
-urllib3==1.26.16
+urllib3==1.26.18
 uvloop==0.17.0
 validators==0.20.0
 wcwidth==0.2.6

requirements/idea-dev-lambda.txt

@@ -1,11 +1,11 @@
 certifi==2023.7.22
 cffi==1.15.1
 charset-normalizer==3.2.0
-cryptography==41.0.3
+cryptography==41.0.4
 idna==3.4
 opensearch-py==2.2.0
 pycparser==2.21
 python-dateutil==2.8.2
 requests==2.31.0
 six==1.16.0
-urllib3==1.26.16
+urllib3==1.26.18

requirements/idea-scheduler.txt

@@ -1,18 +1,19 @@
 aiofiles==23.1.0
 alembic==1.11.1
+annotated-types==0.5.0
 arrow==1.2.3
 banal==1.0.6
 blinker==1.6.2
-boto3==1.28.12
-botocore==1.31.12
+boto3==1.28.44
+botocore==1.31.44
 cacheout==0.14.1
 cachetools==5.3.1
 certifi==2023.7.22
 cffi==1.15.1
 cfn-flip==1.3.0
 charset-normalizer==3.2.0
 click==8.1.6
-cryptography==41.0.3
+cryptography==41.0.4
 dataset==1.6.2
 decorator==5.1.1
 fastcounter==1.1.0
@@ -36,7 +37,8 @@ prometheus-client==0.17.1
 prompt-toolkit==3.0.39
 psutil==5.9.5
 pycparser==2.21
-pydantic==1.10.12
+pydantic==2.3.0
+pydantic-core==2.6.3
 pygments==2.15.1
 pyhocon==0.3.60
 pyjwt==2.8.0
@@ -62,11 +64,11 @@ sqlalchemy==1.4.49
 supervisor==4.2.5
 tomli==2.0.1
 tracerite==1.1.0
-troposphere==4.3.2
+troposphere==4.4.1
 typing-extensions==4.7.1
 tzlocal==5.0.1
 ujson==5.8.0
-urllib3==1.26.16
+urllib3==1.26.18
 uvloop==0.17.0
 validators==0.20.0
 wcwidth==0.2.6

requirements/idea-sdk.in

@@ -4,12 +4,12 @@ pyparsing
 pytz
 tzlocal
 arrow
-pydantic~=1.10.12
+pydantic
 PyYAML
 orjson
 Click
 botocore
-boto3==1.28.12
+boto3==1.28.44
 requests
 requests-aws4auth
 requests-unixsocket