DontReview Garnitin/add gke load testing/v1 #2225

gargnitingoogle · 2024-07-26T21:27:18Z

Description

Link to the issue in case of a bug fix.

NA

Testing details

Manual - NA
Unit tests - NA
Integration tests - NA

codecov · 2024-07-26T21:32:04Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.04%. Comparing base (17414f3) to head (a0b0096).
Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2225      +/-   ##
==========================================
+ Coverage   77.02%   77.04%   +0.01%     
==========================================
  Files         113      113              
  Lines       15910    15918       +8     
==========================================
+ Hits        12255    12264       +9     
+ Misses       3129     3128       -1     
  Partials      526      526

Flag	Coverage Δ
unittests	`77.04% <ø> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

gargnitingoogle · 2024-07-26T21:29:20Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+fi
+
+function validateMachineConfig() {
+  echo "Validiting input parameters ..."


gargnitingoogle · 2024-07-26T21:29:52Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+  local cluster_name=${1}
+  local zone=${2}
+  local node_pool=${3}
+  if [ $(gcloud container node-pools list --cluster=${cluster_name} --zone=${zone} | grep -ow ${node_pool} | wc -l) -gt 0 ] ; then


simplify this with grep -q

gargnitingoogle · 2024-07-26T21:30:59Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+  if ClusterExists ${cluster_name} ; then
+    gcloud container clusters update ${cluster_name} --location=${zone} --workload-pool=${project_id}.svc.id.goog
+  else
+    # gcloud container --project "${project_id}" clusters create ${cluster_name} --zone "${zone}" --cluster-version "${cluster_version}" --workload-pool=${project_id}.svc.id.goog


remove commented code

gargnitingoogle · 2024-07-26T21:32:10Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+  echo "Enabling/disabling csi add-on ..."
+  # By default, disable the managed csi driver.
+  if ${useCustomCsiDriver}; then
+    # gcloud -q container clusters update ${cluster_name} \


re-enable this

gargnitingoogle · 2024-07-26T21:32:29Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+  gcloud container clusters get-credentials ${cluster_name} --location=${zone}
+  kubectl create namespace ${appnamespace}
+  kubectl create serviceaccount ${ksa} --namespace ${appnamespace}
+  for workload_bucket in ${buckets} ; do


add code to get buckets from somewhere.

gargnitingoogle · 2024-07-26T21:35:00Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+}
+
+# validateMachineConfig ${machine_type} ${num_nodes} ${num_ssd}
+# installDependencies


re-enable all these disabled steps.

gargnitingoogle · 2024-07-26T21:36:51Z

perfmetrics/scripts/testing_on_gke/utils/utils.py

+def get_cpu(pod_name: str, start: str, end: str) -> Tuple[float, float]:
+    # for some reason, the mash filter does not always work, so we fetch all the metrics for all the pods and filter later.
+    result = subprocess.run(["mash", "--namespace=cloud_prod", "--output=csv", 
+                             f"Query(Fetch(Raw('cloud.kubernetes.K8sContainer', 'kubernetes.io/container/cpu/core_usage_time'), {{'project': '927584127901'}})| Window(Rate('10m'))| GroupBy(['pod_name', 'container_name'], Max()), TimeInterval('{start}', '{end}'), '5s')"], 


put the project-number back to the original and change it by code during runtime and then revert it back when done during runtime.

gargnitingoogle · 2024-07-26T21:37:31Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+function updateMachineTypeInPodConfigs() {
+  for file in ${gke_testing_dir}/examples/fio/loading-test/values.yaml ${gke_testing_dir}/examples/dlio/unet3d-loading-test/values.yaml ; do
+    test -f ${file}
+    sed -i -E "s/nodeType: [0-9a-z_-]+$/nodeType: ${machine_type}/g" ${file}


add code to revert this back

gargnitingoogle · 2024-07-26T21:37:42Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+  for file in ${gke_testing_dir}/examples/fio/loading-test/values.yaml ${gke_testing_dir}/examples/dlio/unet3d-loading-test/values.yaml ; do
+    test -f ${file}
+    # sed -i -E "s/mountOptions: [0-9a-zA-Z,\:\"-_]+$/mountOptions: \"${gcsfuse_mount_options}\"/g" ${file}
+    sed -i -E "s/mountOptions:[ \t]*\"?[a-zA-Z0-9,:_-]+\"? *$/mountOptions: \"${gcsfuse_mount_options}\"/g" "${file}"


add code to revert this back when done

gargnitingoogle · 2024-07-27T04:27:38Z

perfmetrics/scripts/testing_on_gke/utils/utils.py

+    return utc_timestamp_string
+
+def standard_timestamp(timestamp: int) -> str:
+    return timestamp.split('.')[0].replace('T', ' ') + " UTC"


insert newline after this line

gargnitingoogle · 2024-07-27T04:28:16Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

@@ -0,0 +1,425 @@
+#!/bin/bash
+


Add a header and a help option at the top.

gargnitingoogle commented Jul 26, 2024

View reviewed changes

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 2958a3c to ad97f82 Compare July 29, 2024 10:47

gargnitingoogle commented Jul 29, 2024

View reviewed changes

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch 7 times, most recently from ff7ee55 to 478921c Compare August 6, 2024 09:53

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch 9 times, most recently from 1e4353a to b6a0e76 Compare August 16, 2024 14:45

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch 9 times, most recently from b62b517 to d008a84 Compare August 22, 2024 11:53

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 0daf723 to 6dbb552 Compare August 26, 2024 09:05

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from 53def3b to d423275 Compare December 3, 2024 06:09

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from ace404a to 841048c Compare December 3, 2024 06:09

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from d423275 to 711aa0f Compare December 4, 2024 06:09

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 841048c to c1f5947 Compare December 4, 2024 06:09

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from 711aa0f to 789b5f4 Compare December 5, 2024 06:09

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from c1f5947 to db9edf9 Compare December 5, 2024 06:09

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from 789b5f4 to a3feb0d Compare December 6, 2024 06:09

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from db9edf9 to 1ea03c4 Compare December 6, 2024 06:09

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from a3feb0d to 8294243 Compare December 6, 2024 08:53

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 1ea03c4 to fab819c Compare December 6, 2024 08:53

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from 8294243 to dcfb13b Compare December 6, 2024 10:13

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from fab819c to 137aed1 Compare December 6, 2024 10:13

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from dcfb13b to 924758e Compare December 9, 2024 06:20

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 137aed1 to 8c4d392 Compare December 9, 2024 06:20

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from 924758e to f2f6b68 Compare March 6, 2025 09:36

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 8c4d392 to e3ce844 Compare March 6, 2025 09:36

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from f2f6b68 to 44243a3 Compare March 9, 2025 06:28

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from e3ce844 to e8828d2 Compare March 9, 2025 06:28

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from 44243a3 to 7c135ea Compare March 10, 2025 05:04

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from e8828d2 to 6f6bf16 Compare March 10, 2025 05:04

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from 7c135ea to 5d01790 Compare March 10, 2025 05:32

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 6f6bf16 to 45b39ce Compare March 10, 2025 05:32

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from 5d01790 to 8d14955 Compare March 10, 2025 12:06

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 45b39ce to 02f144c Compare March 10, 2025 12:06

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from 8d14955 to b9dbc4e Compare March 10, 2025 12:11

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 02f144c to 06cc29b Compare March 10, 2025 12:11

gargnitingoogle added 2 commits March 11, 2025 11:48

expose force_update_gcsfuse_code

0678d58

empty commit

3496cd4

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/enhancements/remove-mash-dependency branch from b9dbc4e to b4bd35d Compare March 11, 2025 11:48

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 06cc29b to 3496cd4 Compare March 11, 2025 11:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DontReview Garnitin/add gke load testing/v1 #2225

DontReview Garnitin/add gke load testing/v1 #2225

gargnitingoogle commented Jul 26, 2024

codecov bot commented Jul 26, 2024 •

edited

Loading

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 27, 2024

gargnitingoogle Jul 27, 2024

DontReview Garnitin/add gke load testing/v1 #2225

Are you sure you want to change the base?

DontReview Garnitin/add gke load testing/v1 #2225

Conversation

gargnitingoogle commented Jul 26, 2024

Description

Link to the issue in case of a bug fix.

Testing details

codecov bot commented Jul 26, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jul 26, 2024 •

edited

Loading