Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[testing-on-gke part 7] Code clean-up and readability #2369

Closed
wants to merge 65 commits into from
Closed

[testing-on-gke part 7] Code clean-up and readability #2369

wants to merge 65 commits into from

Conversation

gargnitingoogle
Copy link
Collaborator

@gargnitingoogle gargnitingoogle commented Aug 20, 2024 •
edited
Loading

Description

  1. Restructures and formats code for better reuse and readability.
  2. Supports GCE VMs for monitoring as mash can't be installed in GCE VM , so this uses google-monitoring time-series data for getting the cpu/memory metrics.
  3. Adds many reliability and usability improvements in the setup.

This is on top of #2482. it is followed up in #2322 .

Link to the issue in case of a bug fix.

NA

Testing details

  1. Manual - NA
  2. Unit tests - NA
  3. Integration tests - NA

@codecov Codecov
Copy link

codecov bot commented Aug 20, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.07%. Comparing base (20e38a7) to head (adedeeb).

Additional details and impacted files 
@@                                 Coverage Diff                                 @@
##           garnitin/add-gke-load-testing/fix-for-gce-vm-v1    #2369      +/-   ##
===================================================================================
+ Coverage                                            79.04%   79.07%   +0.03%     
===================================================================================
  Files                                                  105      105              
  Lines                                                11577    11577              
===================================================================================
+ Hits                                                  9151     9155       +4     
+ Misses                                                1961     1958       -3     
+ Partials                                               465      464       -1
Flag Coverage Δ
unittests 79.07% <ø> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@gargnitingoogle gargnitingoogle mentioned this pull request Aug 20, 2024
Merged
@gargnitingoogle gargnitingoogle marked this pull request as ready for review August 20, 2024 10:40
@gargnitingoogle gargnitingoogle requested review from Tulsishah and a team as code owners August 20, 2024 10:40
@gargnitingoogle gargnitingoogle requested review from ankitaluthra1 and removed request for a team August 20, 2024 10:40
@gargnitingoogle gargnitingoogle changed the title Code clean-up and readability [testing-on-gke part 7] Code clean-up and readability Aug 20, 2024
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/fix-for-gce-vm-v1 branch 2 times, most recently from a4f79c9 to 20e38a7 Compare August 21, 2024 11:53
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/code-cleanup/v1 branch from 285bb73 to f3fc062 Compare August 21, 2024 11:53
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/code-cleanup/v1 branch from adedeeb to 30ef2d1 Compare August 22, 2024 10:26
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/fix-for-gce-vm-v1 branch from 20e38a7 to c4a968f Compare August 22, 2024 11:17
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/code-cleanup/v1 branch from f618a36 to adc57a8 Compare August 22, 2024 12:53
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/fix-for-gce-vm-v1 branch from c4a968f to c0d2c12 Compare August 22, 2024 12:55
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/fix-for-gce-vm-v1 branch from c0d2c12 to 327a9b4 Compare August 27, 2024 02:57
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/code-cleanup/v1 branch from adc57a8 to 4edbcc5 Compare August 27, 2024 02:57
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/fix-for-gce-vm-v1 branch from 327a9b4 to e8a043b Compare August 27, 2024 02:59
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/code-cleanup/v1 branch 2 times, most recently from 4e73874 to 982cf7f Compare August 27, 2024 03:13
@gargnitingoogle gargnitingoogle changed the title [testing-on-gke part 7] Code clean-up and readability [testing-on-gke part 6] Code clean-up and readability Aug 28, 2024
@gargnitingoogle gargnitingoogle mentioned this pull request Aug 28, 2024
Merged
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/fix-for-gce-vm-v1 branch from 7abda81 to e90a0e5 Compare August 29, 2024 04:21
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/code-cleanup/v1 branch from 6570892 to 7cc3351 Compare August 29, 2024 04:22
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/fix-for-gce-vm-v1 branch from e90a0e5 to 73467ec Compare September 2, 2024 05:27
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/code-cleanup/v1 branch from 7cc3351 to 5f63247 Compare September 2, 2024 05:27
@gargnitingoogle
Improvements in run-script
33c6a11 
borrow fixes from merge-branches
fix bug in run-script

1. Fix the logic supporting relative paths of
   src_dir, gcsfuse_src_dir, csi_src_dir etc.
2. Fix the logging and checking of the above
   paths existing.

improve logs in run-script
remove an unused bit of code
count pods with error in status as completed
print use_custom_csi_driver
publicize gcsfuse_branch env var
install monitoring even if mash is installed
@gargnitingoogle
Updated comments in workloads json
7f734a1
@gargnitingoogle
fix gcloud installation command
143d225
@gargnitingoogle
Add file-cache:max-size-mb:-1 to all workload configs
b6cdfd9
@gargnitingoogle
fix handling of only_parse=false
d98e42f
@gargnitingoogle
add resource limits/requests for c3-standard-176
956aea7
@gargnitingoogle
add resource config for machine-type c2-standard-176-lssd
2ceb813
@gargnitingoogle
add test for resource config
29ee495
@gargnitingoogle
Ensure that dependency installation worked
1688257
@gargnitingoogle
insert project-id in gcloud commands
7dbad10
@gargnitingoogle
Fix naming of package_bucket
d24d419
@gargnitingoogle
Remove references to read_cache
2213f20 
Remove references to old branch read_cache_release
as well as use appropriate names for fio job files rather than
read_cache_test.fio .
@gargnitingoogle
fix in existing cluster/node-pool utilities
95cd8ef
@gargnitingoogle
improve log message at waiting for pods
2086965
@gargnitingoogle
insert cluster_name in custom-driver-image-name
81925f8 
Insert cluster_name in produced custom-driver
image-name for disambiguation in case a single user
is using multiple clusters with two or more different
gcsfuse/csi driver codebases. In the current code,
both end up using the same CSI driver binary. This
change creates different binaries for such cases,
allowing different set of tests to work.
@gargnitingoogle
improve logs in run-script
8da0ad2
@gargnitingoogle
skip failing fio outputs scenarios and print debug logs
488a693
@gargnitingoogle
improve machine configuration constraints
06439a9
@gargnitingoogle
add utilities for project id<->number translations
cbd5f17
@gargnitingoogle
Handle only one out of project_id, project_number passed
f02705f
@gargnitingoogle
fix formatting couple of files
ae24f4e
@gargnitingoogle
Move bucket-permission logic to python
24f7167 
Move logic to add 'roles/storage.objectUser'
from bash script to python for more
robustness. It accurately parses the
workload file for relevant buckets, and
add GCS access permissions only for
those buckets in the current run.
@gargnitingoogle
Added logs of successful output
1859240
@gargnitingoogle
address a review comment
110d337
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/fix-for-gce-vm-v1 branch from 5bf22e2 to 1243c45 Compare September 16, 2024 10:19
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/code-cleanup/v1 branch from f117611 to 110d337 Compare September 16, 2024 10:19
@gargnitingoogle gargnitingoogle marked this pull request as draft September 17, 2024 05:42
@gargnitingoogle gargnitingoogle force-pushed the garnitin/add-gke-load-testing/fix-for-gce-vm-v1 branch from 1243c45 to 1b3cb96 Compare September 17, 2024 07:10
@gargnitingoogle
Copy link
Collaborator Author

Split this PR into #2492, #2489, #2493, #2495, #2496, #2497, #2498, #2499 for ease of review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Tulsishah Tulsishah Awaiting requested review from Tulsishah

@ankitaluthra1 ankitaluthra1 Awaiting requested review from ankitaluthra1 ankitaluthra1 was automatically assigned from GoogleCloudPlatform/gcsfuse-codeowners

@kislaykishore kislaykishore Awaiting requested review from kislaykishore

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@gargnitingoogle