test: retrieve unique package and latest version for YAML generation

[dandhlee]

There were two issues with the original code: the sort -u -t- -k5,5 command was not returning the latest version for each package, and because some package names came with additional dash(es), i.e. python-monitoring-dashboards when python-monitoring exists, some packages were not being regenerated at all. Currently the command returns the following list:

gs://docs-staging-v2/docfx-python-accessapproval-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-aiplatform-0.3.0.tar.gz
gs://docs-staging-v2/docfx-python-apigateway-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-apigeeconnect-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-appengine-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-appenginelogging-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-artifactregistry-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-assuredworkloads-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-automl-0.10.0.tar.gz
gs://docs-staging-v2/docfx-python-bigquery-1.16.0.tar.gz
gs://docs-staging-v2/docfx-python-bigqueryconnection-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-bigquerydatatransfer-0.4.0.tar.gz
gs://docs-staging-v2/docfx-python-bigquerylogging-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-bigquerymigration-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-bigqueryreservation-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-bigquerystorage-0.6.0.tar.gz
gs://docs-staging-v2/docfx-python-bigtable-0.34.0.tar.gz
gs://docs-staging-v2/docfx-python-billingbudgets-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-binaryauthorization-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-certificatemanager-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-cloudasset-0.10.0.tar.gz
gs://docs-staging-v2/docfx-python-cloudbilling-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-cloudbuild-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-cloudchannel-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-clouddebugger-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-clouddeploy-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-clouderrorreporting-0.32.1.tar.gz
gs://docs-staging-v2/docfx-python-cloudfunctions-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-cloudiot-0.3.0.tar.gz
gs://docs-staging-v2/docfx-python-cloudkms-1.1.0.tar.gz
gs://docs-staging-v2/docfx-python-cloudprivatecatalog-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-cloudresourcemanager-0.29.2.tar.gz
gs://docs-staging-v2/docfx-python-cloudscheduler-1.2.0.tar.gz
gs://docs-staging-v2/docfx-python-cloudshell-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-cloudtasks-1.2.0.tar.gz
gs://docs-staging-v2/docfx-python-cloudtrace-0.22.0.tar.gz
gs://docs-staging-v2/docfx-python-common-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-composer-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-compute-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-connectgateway-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-contactcenterinsights-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-container-0.3.0.tar.gz
gs://docs-staging-v2/docfx-python-containeranalysis-0.2.0.tar.gz
gs://docs-staging-v2/docfx-python-datacatalog-0.3.0.tar.gz
gs://docs-staging-v2/docfx-python-dataflow-0.1.1.tar.gz
gs://docs-staging-v2/docfx-python-datafusion-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-datalabeling-0.2.0.tar.gz
gs://docs-staging-v2/docfx-python-datamigration-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-dataplex-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-dataproc-0.5.0.tar.gz
gs://docs-staging-v2/docfx-python-dataqna-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-datastore-1.10.0.tar.gz
gs://docs-staging-v2/docfx-python-datastream-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-dialogflow-0.7.0.tar.gz
gs://docs-staging-v2/docfx-python-dlp-0.12.0.tar.gz
gs://docs-staging-v2/docfx-python-dns-0.30.2.tar.gz
gs://docs-staging-v2/docfx-python-documentai-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-domains-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-essentialcontacts-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-eventarc-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-eventarcpublishing-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-file-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-firestore-1.3.0.tar.gz
gs://docs-staging-v2/docfx-python-gameservices-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-gkehub-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-google-cloud-core-1.4.0.tar.gz
gs://docs-staging-v2/docfx-python-iam-0.2.0.tar.gz
gs://docs-staging-v2/docfx-python-iamcredentials-0.3.0.tar.gz
gs://docs-staging-v2/docfx-python-iamlogging-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-iap-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-ids-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-language-1.3.0.tar.gz
gs://docs-staging-v2/docfx-python-lifesciences-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-livestream-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-logging-1.12.0.tar.gz
gs://docs-staging-v2/docfx-python-managedidentities-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-mediatranslation-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-memcache-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-metastore-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-monitoring-0.32.0.tar.gz
gs://docs-staging-v2/docfx-python-networkconnectivity-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-networkmanagement-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-networksecurity-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-notebooks-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-orgpolicy-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-osconfig-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-oslogin-0.2.0.tar.gz
gs://docs-staging-v2/docfx-python-phishingprotection-0.2.0.tar.gz
gs://docs-staging-v2/docfx-python-policytroubleshooter-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-privateca-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-pubsub-0.43.0.tar.gz
gs://docs-staging-v2/docfx-python-pubsublite-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-recaptchaenterprise-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-recommendationengine-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-recommender-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-redis-0.3.0.tar.gz
gs://docs-staging-v2/docfx-python-resourcesettings-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-retail-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-runtimeconfig-0.29.2.tar.gz
gs://docs-staging-v2/docfx-python-secretmanager-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-securitycenter-0.3.0.tar.gz
gs://docs-staging-v2/docfx-python-servicecontrol-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-servicedirectory-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-servicemanagement-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-serviceusage-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-source-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-spanner-1.10.0.tar.gz
gs://docs-staging-v2/docfx-python-speech-1.1.0.tar.gz
gs://docs-staging-v2/docfx-python-storage-1.17.0.tar.gz
gs://docs-staging-v2/docfx-python-storagetransfer-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-talent-0.3.0.tar.gz
gs://docs-staging-v2/docfx-python-texttospeech-0.5.0.tar.gz
gs://docs-staging-v2/docfx-python-tpu-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-transcoder-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-translate-3.7.1.tar.gz
gs://docs-staging-v2/docfx-python-translation-1.6.0.tar.gz
gs://docs-staging-v2/docfx-python-videointelligence-1.10.0.tar.gz
gs://docs-staging-v2/docfx-python-videostitcher-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-vision-0.38.0.tar.gz
gs://docs-staging-v2/docfx-python-vmmigration-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-vpcaccess-0.1.0.tar.gz
gs://docs-staging-v2/docfx-python-webrisk-0.2.0.tar.gz
gs://docs-staging-v2/docfx-python-websecurityscanner-0.3.0.tar.gz
gs://docs-staging-v2/docfx-python-workflows-0.1.0.tar.gz

Fixing this by retrieving the unique package names, then using that to retrieve the latest version for each tarball. This now retrieves the following list:

gs://docs-staging-v2/docfx-python-accessapproval-1.5.1.tar.gz
gs://docs-staging-v2/docfx-python-aiplatform-1.11.0.tar.gz
gs://docs-staging-v2/docfx-python-apigateway-1.2.1.tar.gz
gs://docs-staging-v2/docfx-python-apigeeconnect-1.2.1.tar.gz
gs://docs-staging-v2/docfx-python-appengine-1.3.2.tar.gz
gs://docs-staging-v2/docfx-python-appenginelogging-1.1.1.tar.gz
gs://docs-staging-v2/docfx-python-artifactregistry-1.1.2.tar.gz
gs://docs-staging-v2/docfx-python-assuredworkloads-1.2.2.tar.gz
gs://docs-staging-v2/docfx-python-automl-2.7.2.tar.gz
gs://docs-staging-v2/docfx-python-bigquery-2.34.2.tar.gz
gs://docs-staging-v2/docfx-python-bigqueryconnection-1.3.4.tar.gz
gs://docs-staging-v2/docfx-python-bigquerydatatransfer-3.6.1.tar.gz
gs://docs-staging-v2/docfx-python-bigquerylogging-1.0.2.tar.gz
gs://docs-staging-v2/docfx-python-bigquerymigration-0.3.1.tar.gz
gs://docs-staging-v2/docfx-python-bigqueryreservation-1.6.1.tar.gz
gs://docs-staging-v2/docfx-python-bigquerystorage-2.12.0.tar.gz
gs://docs-staging-v2/docfx-python-bigtable-2.7.0.tar.gz
gs://docs-staging-v2/docfx-python-billingbudgets-1.6.1.tar.gz
gs://docs-staging-v2/docfx-python-binaryauthorization-1.2.2.tar.gz
gs://docs-staging-v2/docfx-python-certificatemanager-0.1.1.tar.gz
gs://docs-staging-v2/docfx-python-cloudasset-3.8.1.tar.gz
gs://docs-staging-v2/docfx-python-cloudbilling-1.5.1.tar.gz
gs://docs-staging-v2/docfx-python-cloudbuild-3.8.1.tar.gz
gs://docs-staging-v2/docfx-python-cloudchannel-1.7.1.tar.gz
gs://docs-staging-v2/docfx-python-clouddebugger-1.3.1.tar.gz
gs://docs-staging-v2/docfx-python-clouddeploy-1.1.1.tar.gz
gs://docs-staging-v2/docfx-python-clouderrorreporting-1.5.0.tar.gz
gs://docs-staging-v2/docfx-python-cloudfunctions-1.5.2.tar.gz
gs://docs-staging-v2/docfx-python-cloudiot-2.4.1.tar.gz
gs://docs-staging-v2/docfx-python-cloudkms-2.11.1.tar.gz
gs://docs-staging-v2/docfx-python-cloudprivatecatalog-0.5.1.tar.gz
gs://docs-staging-v2/docfx-python-cloudresourcemanager-1.4.1.tar.gz
gs://docs-staging-v2/docfx-python-cloudscheduler-2.6.2.tar.gz
gs://docs-staging-v2/docfx-python-cloudshell-1.3.2.tar.gz
gs://docs-staging-v2/docfx-python-cloudtasks-2.8.1.tar.gz
gs://docs-staging-v2/docfx-python-cloudtrace-1.6.1.tar.gz
gs://docs-staging-v2/docfx-python-common-1.0.1.tar.gz
gs://docs-staging-v2/docfx-python-composer-1.3.1.tar.gz
gs://docs-staging-v2/docfx-python-compute-1.0.0.tar.gz
gs://docs-staging-v2/docfx-python-connectgateway-0.5.1.tar.gz
gs://docs-staging-v2/docfx-python-contactcenterinsights-1.3.1.tar.gz
gs://docs-staging-v2/docfx-python-container-2.10.6.tar.gz
gs://docs-staging-v2/docfx-python-containeranalysis-2.7.2.tar.gz
gs://docs-staging-v2/docfx-python-datacatalog-3.7.1.tar.gz
gs://docs-staging-v2/docfx-python-dataflow-0.4.1.tar.gz
gs://docs-staging-v2/docfx-python-datafusion-1.4.1.tar.gz
gs://docs-staging-v2/docfx-python-datalabeling-1.5.1.tar.gz
gs://docs-staging-v2/docfx-python-datamigration-1.3.1.tar.gz
gs://docs-staging-v2/docfx-python-dataplex-0.2.1.tar.gz
gs://docs-staging-v2/docfx-python-dataproc-4.0.1.tar.gz
gs://docs-staging-v2/docfx-python-dataqna-0.6.2.tar.gz
gs://docs-staging-v2/docfx-python-datastore-2.5.1.tar.gz
gs://docs-staging-v2/docfx-python-datastream-0.4.2.tar.gz
gs://docs-staging-v2/docfx-python-dialogflow-2.12.0.tar.gz
gs://docs-staging-v2/docfx-python-dialogflow-cx-1.9.1.tar.gz
gs://docs-staging-v2/docfx-python-dlp-3.6.2.tar.gz
gs://docs-staging-v2/docfx-python-dns-0.34.0.tar.gz
gs://docs-staging-v2/docfx-python-documentai-1.3.0.tar.gz
gs://docs-staging-v2/docfx-python-domains-1.1.1.tar.gz
gs://docs-staging-v2/docfx-python-essentialcontacts-1.1.1.tar.gz
gs://docs-staging-v2/docfx-python-eventarc-1.4.1.tar.gz
gs://docs-staging-v2/docfx-python-eventarcpublishing-0.1.1.tar.gz
gs://docs-staging-v2/docfx-python-file-1.1.1.tar.gz
gs://docs-staging-v2/docfx-python-firestore-2.3.4.tar.gz
gs://docs-staging-v2/docfx-python-gameservices-1.4.1.tar.gz
gs://docs-staging-v2/docfx-python-gkehub-1.4.1.tar.gz
gs://docs-staging-v2/docfx-python-google-cloud-core-2.2.3.tar.gz
gs://docs-staging-v2/docfx-python-google-cloud-monitoring-metrics-scopes-1.1.1.tar.gz
gs://docs-staging-v2/docfx-python-google-resumable-media-1.3.1.tar.gz
gs://docs-staging-v2/docfx-python-iam-2.5.1.tar.gz
gs://docs-staging-v2/docfx-python-iamcredentials-2.6.1.tar.gz
gs://docs-staging-v2/docfx-python-iamlogging-1.0.1.tar.gz
gs://docs-staging-v2/docfx-python-iap-1.3.1.tar.gz
gs://docs-staging-v2/docfx-python-ids-1.1.1.tar.gz
gs://docs-staging-v2/docfx-python-language-2.4.1.tar.gz
gs://docs-staging-v2/docfx-python-lifesciences-0.5.1.tar.gz
gs://docs-staging-v2/docfx-python-livestream-0.1.1.tar.gz
gs://docs-staging-v2/docfx-python-logging-3.0.0.tar.gz
gs://docs-staging-v2/docfx-python-managedidentities-1.4.1.tar.gz
gs://docs-staging-v2/docfx-python-mediatranslation-0.8.1.tar.gz
gs://docs-staging-v2/docfx-python-memcache-1.3.1.tar.gz
gs://docs-staging-v2/docfx-python-metastore-1.4.1.tar.gz
gs://docs-staging-v2/docfx-python-monitoring-2.9.1.tar.gz
gs://docs-staging-v2/docfx-python-monitoring-dashboards-2.5.1.tar.gz
gs://docs-staging-v2/docfx-python-networkconnectivity-1.3.1.tar.gz
gs://docs-staging-v2/docfx-python-networkmanagement-1.3.1.tar.gz
gs://docs-staging-v2/docfx-python-networksecurity-0.4.1.tar.gz
gs://docs-staging-v2/docfx-python-notebooks-1.2.1.tar.gz
gs://docs-staging-v2/docfx-python-orgpolicy-1.3.1.tar.gz
gs://docs-staging-v2/docfx-python-osconfig-1.11.1.tar.gz
gs://docs-staging-v2/docfx-python-oslogin-2.6.1.tar.gz
gs://docs-staging-v2/docfx-python-phishingprotection-1.5.1.tar.gz
gs://docs-staging-v2/docfx-python-policytroubleshooter-1.4.1.tar.gz
gs://docs-staging-v2/docfx-python-privateca-1.2.4.tar.gz
gs://docs-staging-v2/docfx-python-pubsub-2.9.0.tar.gz
gs://docs-staging-v2/docfx-python-pubsublite-1.4.1.tar.gz
gs://docs-staging-v2/docfx-python-recaptchaenterprise-1.6.1.tar.gz
gs://docs-staging-v2/docfx-python-recommendationengine-0.6.1.tar.gz
gs://docs-staging-v2/docfx-python-recommender-2.7.2.tar.gz
gs://docs-staging-v2/docfx-python-redis-2.7.1.tar.gz
gs://docs-staging-v2/docfx-python-resourcesettings-1.3.1.tar.gz
gs://docs-staging-v2/docfx-python-retail-1.4.1.tar.gz
gs://docs-staging-v2/docfx-python-runtimeconfig-0.33.0.tar.gz
gs://docs-staging-v2/docfx-python-secretmanager-2.9.1.tar.gz
gs://docs-staging-v2/docfx-python-securitycenter-1.10.0.tar.gz
gs://docs-staging-v2/docfx-python-servicecontrol-1.3.1.tar.gz
gs://docs-staging-v2/docfx-python-servicedirectory-1.3.1.tar.gz
gs://docs-staging-v2/docfx-python-servicemanagement-1.2.1.tar.gz
gs://docs-staging-v2/docfx-python-serviceusage-1.3.1.tar.gz
gs://docs-staging-v2/docfx-python-source-1.2.2.tar.gz
gs://docs-staging-v2/docfx-python-spanner-3.13.0.tar.gz
gs://docs-staging-v2/docfx-python-speech-2.13.1.tar.gz
gs://docs-staging-v2/docfx-python-storage-2.1.0.tar.gz
gs://docs-staging-v2/docfx-python-storagetransfer-1.2.1.tar.gz
gs://docs-staging-v2/docfx-python-talent-2.5.1.tar.gz
gs://docs-staging-v2/docfx-python-texttospeech-2.10.2.tar.gz
gs://docs-staging-v2/docfx-python-tpu-1.3.2.tar.gz
gs://docs-staging-v2/docfx-python-transcoder-1.3.1.tar.gz
gs://docs-staging-v2/docfx-python-translate-3.7.2.tar.gz
gs://docs-staging-v2/docfx-python-translation-3.6.1.tar.gz
gs://docs-staging-v2/docfx-python-videointelligence-2.6.1.tar.gz
gs://docs-staging-v2/docfx-python-videostitcher-0.1.1.tar.gz
gs://docs-staging-v2/docfx-python-vision-2.7.1.tar.gz
gs://docs-staging-v2/docfx-python-vmmigration-1.1.1.tar.gz
gs://docs-staging-v2/docfx-python-vpcaccess-1.3.1.tar.gz
gs://docs-staging-v2/docfx-python-webrisk-1.7.1.tar.gz
gs://docs-staging-v2/docfx-python-websecurityscanner-1.7.1.tar.gz
gs://docs-staging-v2/docfx-python-workflows-1.6.1.tar.gz

I've verified manually that there are no packages dropped compared to the bucket, as well as that each of these are the latest version of each tarball.

Fixes #189.

• Tests pass

[tbpg]

LGTM.

[tbpg]

🤯 TIL about sort -V.