Support Amazon Linux 2023 #2692

himani2411 · 2024-04-08T21:09:49Z

Description of changes

PR to run Install and Config kitchen tests after changes from [POC] Add support for Amazon Linux 2023 #2343
See commit descriptions for details

Tests

AMIs build are successful. All integration tests in develop.yaml have been run.
Only the following 6 tests are failing:

test-suites:
  basic:
    test_essential_features.py::test_essential_features:
      dimensions:
      - instances:
        - c5.xlarge
        oss:
        - alinux2023
        regions:
        - us-east-1
        schedulers:
        - slurm
  createami:
    test_createami.py::test_build_image_custom_components:
      dimensions:
      - instances:
        - m6g.xlarge
        oss:
        - alinux2023
        regions:
        - us-west-1
      - instances:
        - c5.xlarge
        oss:
        - alinux2023
        regions:
        - eu-north-1
  iam:
    test_iam_image.py::test_iam_roles:
      dimensions:
      - instances:
        - c5.xlarge
        oss:
        - alinux2023
        regions:
        - us-west-2
  networking:
    test_cluster_networking.py::test_cluster_in_no_internet_subnet:
      dimensions:
      - instances:
        - c5.xlarge
        oss:
        - alinux2023
        regions:
        - us-east-1
        schedulers:
        - slurm
  trainium:
    test_trainium.py::test_trainium:
      dimensions:
      - oss:
        - alinux2023
        regions:
        - us-west-2
        schedulers:
        - slurm

References

CLI PR Support Amazon Linux 2023 aws-parallelcluster#6223

Checklist

Make sure you are pointing to the right branch.
If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
Check all commits' messages are clear, describing what and why vs how.
Make sure to have added unit tests or integration tests to cover the new/modified code.
Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

codecov · 2024-04-22T15:17:44Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.48%. Comparing base (7d414fe) to head (ecbd953).
Report is 76 commits behind head on develop.

❗ Current head ecbd953 differs from pull request most recent head 5552b26

Please upload reports for the commit 5552b26 to get more accurate results.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #2692   +/-   ##
========================================
  Coverage    76.48%   76.48%           
========================================
  Files           22       22           
  Lines         2220     2220           
========================================
  Hits          1698     1698           
  Misses         522      522

Flag	Coverage Δ
unittests	`76.48% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

himani2411

I am not sure whether we require this specific commit or not as the [TEMP] taggged commits were something that I was adding to find a solution to few errors that I was getting

84e0d19

7cf55fd

cookbooks/aws-parallelcluster-computefleet/metadata.rb

cookbooks/aws-parallelcluster-environment/kitchen.environment-install.yml

cookbooks/aws-parallelcluster-environment/resources/ec2_udev_rules/ec2_udev_rules_alinux2023.rb

cookbooks/aws-parallelcluster-environment/test/controls/efs_spec.rb

cookbooks/aws-parallelcluster-environment/test/controls/lustre_spec.rb

cookbooks/aws-parallelcluster-slurm/resources/dns_domain/dns_domain_alinux2023.rb

gmarciani · 2024-06-07T12:23:17Z

cookbooks/aws-parallelcluster-slurm/resources/mysql_client/partial/_setup_rhel_based.rb

@@ -49,6 +49,8 @@ def package_platform
    platform_version = node['platform_version'].to_i
    if platform_version == 2
      platform_version = 7
+    elsif platform_version == 2023


Bad practice: redefinition of the same variable with different meaning.
Here we should have platform_version and mysql_platform_version

Here we are using the platform_version is just the location in our s3 bucket where we archive the packages we want. Similar to what we do for Cinc-client.

archives/mysql/el/9/x86_64/mysql-community-client-*tar.gz

Sure, the bad practice is doing:

platform_version = define platform version with meaning 1 if platform_version == 2 platform_version = 7 use platform version with meaning 2

Not a blocking comment because the practice is not introduced in this PR, but better to address it at least in a follow up PR

...ooks/aws-parallelcluster-slurm/resources/slurm_dependencies/slurm_dependencies_alinux2023.rb

gmarciani · 2024-06-07T12:25:31Z

kitchen/kitchen.run-hook.sh

@@ -37,6 +37,10 @@ if [ -e "${SCRIPT}" ]; then
        TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600") \
        && curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/instance-id
      ')
+      echo "Install libxcrypt-compat dmidecode package by using SSH key: ${KITCHEN_SSH_KEY_PATH}"


If this is something required only by AL2023, let's add a comment saying that.
If this is not, why adding it now?

Since this is in test code, can we address this later?

util/cinc-install.sh

enrico-usai · 2024-06-11T08:09:24Z

...ws-parallelcluster-environment/files/alinux2023/network_interfaces/configure_nw_interface.sh

+configured_ip=`nmcli -t -f IP4.ADDRESS device show ${DEVICE_NAME} | cut -f2 -d':'`
+if [ -z "${configured_ip}" ]; then
+  # Setup connection method to "manual", configure ip address and gateway, only if not already configured.
+  sudo nmcli connection modify "${con_name}" ipv4.method manual ipv4.addresses ${DEVICE_IP_ADDRESS}/${CIDR_PREFIX_LENGTH} ipv4.gateway ${GW_IP_ADDRESS}


According to this comment nmcli is not supported: https://github.com/aws/aws-parallelcluster/pull/6223/files#r1630201744

After testing, Amazon Linux 2023 does not require this script because amazon-ec2-net-utils is pre-installed in AL2023 and handles multi-nics instances properly.

test_efa has passed on multi-nics instances

Amazon Linux 2023 does not require this script
So we are removing the whole file, right?

cookbooks/aws-parallelcluster-platform/recipes/install/enable_services.rb

cookbooks/aws-parallelcluster-environment/resources/ec2_udev_rules/ec2_udev_rules_alinux2023.rb

cookbooks/aws-parallelcluster-environment/kitchen.environment-install.yml

cookbooks/aws-parallelcluster-environment/test/controls/efs_spec.rb

cookbooks/aws-parallelcluster-slurm/resources/dns_domain/dns_domain_alinux2023.rb

cookbooks/aws-parallelcluster-environment/test/controls/lustre_spec.rb

hanwen-pcluste · 2024-06-11T00:35:11Z

cookbooks/aws-parallelcluster-environment/resources/cloudwatch/cloudwatch_alinux2023.rb

+action :cloudwatch_prerequisite do
+  package "gnupg2-full" do
+    options '--allowerasing'
+    retries 3
+    retry_delay 5
+  end
+end


This change is from https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2022.0.20220824.html

cookbooks/aws-parallelcluster-platform/spec/unit/recipes/enable_services_spec.rb

cookbooks/aws-parallelcluster-platform/test/controls/enable_services_spec.rb

…flict on Docker but not on EC2

* Changing Package repos test to skip checking Repo name for epel

https://docs.aws.amazon.com/linux/al2023/release-notes/support-info-by-package.html

…lation

Inspec test has not supported Amazon Linux 2023. https://docs.chef.io/inspec/platforms/. Therefore, this commit disable the check which was generating false errors

Running this recipe on Alinux 2023 docker generates false failure: https://github.com/aws/aws-parallelcluster-cookbook/actions/runs/9373643185/job/25807894209?pr=2692 Signed-off-by: Hanwen <hanwenli@amazon.com>

Signed-off-by: Hanwen <hanwenli@amazon.com>

rsyslog is required to have Amazon Linux 2023 writes to messages log. The messages log is uploaded to CloudWatch. Therefore, this commits move the installation and start of rsyslog to CloudWatch recipe and removes the standalone recipe to enable rsyslog. Signed-off-by: Hanwen <hanwenli@amazon.com>

amazon-ec2-net-utils is pre-installed in AL2023 and handles multi-nics instances properly Signed-off-by: Hanwen <hanwenli@amazon.com>

Signed-off-by: Hanwen <hanwenli@amazon.com>

gmarciani · 2024-06-12T14:19:39Z

cookbooks/aws-parallelcluster-platform/recipes/install/cuda.rb

@@ -50,7 +50,9 @@
  cwd '/tmp'
  code <<-CUDA
    set -e
-    ./cuda.run --silent --toolkit --samples
+    mkdir /cuda-install
+    ./cuda.run --silent --toolkit --samples --tmpdir=/cuda-install


/cuda-install is meant to be a temporary directory, so we should create it under /tmp.
That said it's fine to use recursive deletion of that folder since it's a temp directory created by us and not meant ot be used by the user.

/tmp on some OSes has size limit. That's why we had to change the directory

Then please add a comment on top of it to explain why we are not using tmp dir.
However, we should be in control of the tmp dir size and adjust it to our needs.
May you please track in the backlog the possibility to control the /tmp size as a separate partition (that is also a storage best practice)?

Done, added a comment

himani2411 added skip-changelog-update 3.x labels Apr 8, 2024

himani2411 requested review from a team as code owners April 8, 2024 21:09

himani2411 force-pushed the al2023 branch 3 times, most recently from 6502d28 to cf8cc4d Compare April 15, 2024 17:24

himani2411 force-pushed the al2023 branch 2 times, most recently from b587772 to 86792ad Compare April 22, 2024 17:02

himani2411 mentioned this pull request Apr 22, 2024

Upgrade Cinc to version 18.4.12 #2713

Merged

himani2411 force-pushed the al2023 branch 3 times, most recently from 922c9b3 to e2f924c Compare May 1, 2024 18:42

himani2411 force-pushed the al2023 branch from e2f924c to c9fd702 Compare May 8, 2024 12:00

This was referenced May 23, 2024

[POC] Add support for Amazon Linux 2023 #2343

Closed

Feature request: AmazonLinux 2023 (al2023) support aws/aws-parallelcluster#5214

Closed

hanwen-pcluste force-pushed the al2023 branch from c9fd702 to f4f959f Compare June 3, 2024 18:28

hanwen-pcluste changed the title ~~[DRAFT][POC] Supporting AL 2023.4.1~~ Support Amazon Linux 2023 Jun 3, 2024

hanwen-pcluste mentioned this pull request Jun 4, 2024

Support Amazon Linux 2023 aws/aws-parallelcluster#6223

Merged

hanwen-pcluste force-pushed the al2023 branch 3 times, most recently from dcaf56d to c63edb9 Compare June 5, 2024 14:06

himani2411 commented Jun 5, 2024

View reviewed changes

gmarciani reviewed Jun 7, 2024

View reviewed changes

hanwen-pcluste force-pushed the al2023 branch from c63edb9 to 1ecd84d Compare June 7, 2024 14:52

enrico-usai reviewed Jun 11, 2024

View reviewed changes

hanwen-pcluste force-pushed the al2023 branch from 1ecd84d to 3f4cacf Compare June 11, 2024 15:42

hanwen-pcluste reviewed Jun 11, 2024

View reviewed changes

hanwen-pcluste force-pushed the al2023 branch from 3f4cacf to 99260c8 Compare June 11, 2024 21:04

Himani Deshpande and others added 25 commits June 11, 2024 14:14

[AL2023]Add Comment for installing full featured packages

0517b0d

[AL2]removing duplicate functions in AL2022

be7ce03

[AL2023]CookStyle change

7ab5ed8

[AL2023] Adding Github Actions for running Kitchen tests for Al2023

e62711e

[AL2023] Unit tests for Nvidia Driver

ee0f2b2

[AL2023] Installing the full versions of packages as some of them con…

f25ad05

…flict on Docker but not on EC2

[AL2023] yum-utils is not supported in AL2023

b346ad5

* Changing Package repos test to skip checking Repo name for epel

[AL2023] Supporting AL2023 for Disable Sudo Access

8065007

[AL2023] Adding packages which are now supported with differnt names

6a6c076

https://docs.aws.amazon.com/linux/al2023/release-notes/support-info-by-package.html

[AL2023] Supporting Cluster User for Al2023

3b21c22

[AL2023]Adding Amazon in test metadata

419a273

[AL2023] Install dmidecode

10faecd

[AL2023]Update metadata for AL2023

c424659

[AL2023] Adding Guards for NVIDIA

960ff9f

[AL2023] Support Cuda Installation by using root directory for instal…

51930ac

…lation

[AL2023] Adding installation of rsyslog and enabling the service

6351b86

[AL2023] Supporting system authentication for AD

9136993

[AL2023] Remove check to assert service running

07a8dd2

Inspec test has not supported Amazon Linux 2023. https://docs.chef.io/inspec/platforms/. Therefore, this commit disable the check which was generating false errors

[AL2023] Avoid openssh recipe on Amazon Linux 2023 + docker

d42a555

Running this recipe on Alinux 2023 docker generates false failure: https://github.com/aws/aws-parallelcluster-cookbook/actions/runs/9373643185/job/25807894209?pr=2692 Signed-off-by: Hanwen <hanwenli@amazon.com>

[AL2023] mock directory when running tests on docker

89d077c

Signed-off-by: Hanwen <hanwenli@amazon.com>

[AL2023] remove prehook

b58c61a

Signed-off-by: Hanwen <hanwenli@amazon.com>

Remove support information in metadata

33fd1d0

Signed-off-by: Hanwen <hanwenli@amazon.com>

[AL2023] Remove network interfaces configuration

9286e4c

amazon-ec2-net-utils is pre-installed in AL2023 and handles multi-nics instances properly Signed-off-by: Hanwen <hanwenli@amazon.com>

[AL2023] Remove outdated packages

1357fd7

Signed-off-by: Hanwen <hanwenli@amazon.com>

hanwen-pcluste force-pushed the al2023 branch from 99260c8 to 1357fd7 Compare June 11, 2024 21:41

Merge branch 'develop' into al2023

5552b26

gmarciani reviewed Jun 12, 2024

View reviewed changes

gmarciani approved these changes Jun 12, 2024

View reviewed changes

hanwen-pcluste merged commit 095f581 into aws:develop Jun 12, 2024
29 of 31 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Amazon Linux 2023 #2692

Support Amazon Linux 2023 #2692

himani2411 commented Apr 8, 2024 •

edited by hanwen-pcluste

Loading

codecov bot commented Apr 22, 2024 •

edited

Loading

himani2411 left a comment •

edited

Loading

gmarciani Jun 7, 2024

himani2411 Jun 10, 2024 •

edited

Loading

gmarciani Jun 12, 2024

gmarciani Jun 7, 2024

hanwen-pcluste Jun 11, 2024

enrico-usai Jun 11, 2024

hanwen-pcluste Jun 11, 2024

gmarciani Jun 12, 2024

hanwen-pcluste Jun 11, 2024

gmarciani Jun 12, 2024

hanwen-pcluste Jun 12, 2024

gmarciani Jun 12, 2024

hanwen-pcluste Jun 12, 2024

Support Amazon Linux 2023 #2692

Support Amazon Linux 2023 #2692

Conversation

himani2411 commented Apr 8, 2024 • edited by hanwen-pcluste Loading

Description of changes

Tests

References

Checklist

codecov bot commented Apr 22, 2024 • edited Loading

Codecov Report

himani2411 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

himani2411 Jun 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

himani2411 commented Apr 8, 2024 •

edited by hanwen-pcluste

Loading

codecov bot commented Apr 22, 2024 •

edited

Loading

himani2411 left a comment •

edited

Loading

himani2411 Jun 10, 2024 •

edited

Loading