Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

{CI} Put the cloud module at the end of the serial execution to avoid affecting other tests. #26824

Merged
merged 3 commits into from
Jul 10, 2023

Conversation

wangzelin007
Copy link
Member

@wangzelin007 wangzelin007 commented Jul 6, 2023

Related command

Description

Now the test is to dynamically allocate the modules that need to be tested for each instance.
PR #26785 adds a module graphservices. Due to the adjustment of dynamic allocation, on instance 7, we will test the cloud and acr modules in turn. There is a test test_get_docker_credentials in acr that will fail, and the error is as follows:

E ValueError: OIDC Discovery failed on https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/v2.0/.well-known/openid-configuration. HTTP status: 400, Error: {"error":"invalid_tenant","error_description":"AADSTS900021: Requested tenant identifier '00000000-0000-0000-0000-000000000000' is not valid. Tenant identifiers may not be an empty GUID.\r\nTrace ID: 4f691ff2-1fd0-4261-ab70-e1bf37b54600\r\nCorrelation ID: 5ee5b4ef-52f7-48d6-8e24-351d067c7bac\r\nTimestamp: 2023-07-05 08:16:30Z","error_codes":[900021],"timestamp":"2023-07-05 08:16:30Z","trace_id":"4f691ff2-1fd0-4261-ab70-e1bf37b54600","correlation_id":"5ee5b4ef-52f7-48d6-8e24-351d067c7bac"}

During handling of the above exception, another exception occurred:
E ValueError: Unable to get authority configuration for https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000. Authority would typically be in a format of https://login.microsoftonline.com/your_tenant Also please double check your tenant name or GUID is correct.

The reason is that after the cloud test is completed, some changes have been made and are not restored in tearDown. Since the specific reason is not yet determined, the workaround is to put the cloud module at the end for testing to eliminate the impact.

Test screenshot:
image

TODO:
Find the root cause of the failure and modify the relevant test code.

Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature


This checklist is used to make sure that common guidelines for a pull request are followed.

@azure-client-tools-bot-prd
Copy link

azure-client-tools-bot-prd bot commented Jul 6, 2023

️✔️AzureCLI-FullTest
️✔️acr
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️acs
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️advisor
️✔️latest
️✔️3.10
️✔️3.9
️✔️ams
️✔️latest
️✔️3.10
️✔️3.9
️✔️apim
️✔️latest
️✔️3.10
️✔️3.9
️✔️appconfig
️✔️latest
️✔️3.10
️✔️3.9
️✔️appservice
️✔️latest
️✔️3.10
️✔️3.9
️✔️aro
️✔️latest
️✔️3.10
️✔️3.9
️✔️backup
️✔️latest
️✔️3.10
️✔️3.9
️✔️batch
️✔️latest
️✔️3.10
️✔️3.9
️✔️batchai
️✔️latest
️✔️3.10
️✔️3.9
️✔️billing
️✔️latest
️✔️3.10
️✔️3.9
️✔️botservice
️✔️latest
️✔️3.10
️✔️3.9
️✔️cdn
️✔️latest
️✔️3.10
️✔️3.9
️✔️cloud
️✔️latest
️✔️3.10
️✔️3.9
️✔️cognitiveservices
️✔️latest
️✔️3.10
️✔️3.9
️✔️config
️✔️latest
️✔️3.10
️✔️3.9
️✔️configure
️✔️latest
️✔️3.10
️✔️3.9
️✔️consumption
️✔️latest
️✔️3.10
️✔️3.9
️✔️container
️✔️latest
️✔️3.10
️✔️3.9
️✔️core
️✔️2018-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️cosmosdb
️✔️latest
️✔️3.10
️✔️3.9
️✔️databoxedge
️✔️2019-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️dla
️✔️latest
️✔️3.10
️✔️3.9
️✔️dls
️✔️latest
️✔️3.10
️✔️3.9
️✔️dms
️✔️latest
️✔️3.10
️✔️3.9
️✔️eventgrid
️✔️latest
️✔️3.10
️✔️3.9
️✔️eventhubs
️✔️latest
️✔️3.10
️✔️3.9
️✔️feedback
️✔️latest
️✔️3.10
️✔️3.9
️✔️find
️✔️latest
️✔️3.10
️✔️3.9
️✔️hdinsight
️✔️latest
️✔️3.10
️✔️3.9
️✔️identity
️✔️latest
️✔️3.10
️✔️3.9
️✔️iot
️✔️2019-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️keyvault
️✔️2018-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️kusto
️✔️latest
️✔️3.10
️✔️3.9
️✔️lab
️✔️latest
️✔️3.10
️✔️3.9
️✔️managedservices
️✔️latest
️✔️3.10
️✔️3.9
️✔️maps
️✔️latest
️✔️3.10
️✔️3.9
️✔️marketplaceordering
️✔️latest
️✔️3.10
️✔️3.9
️✔️monitor
️✔️latest
️✔️3.10
️✔️3.9
️✔️mysql
️✔️latest
️✔️3.10
️✔️3.9
️✔️netappfiles
️✔️latest
️✔️3.10
️✔️3.9
️✔️network
️✔️2018-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️policyinsights
️✔️latest
️✔️3.10
️✔️3.9
️✔️privatedns
️✔️latest
️✔️3.10
️✔️3.9
️✔️profile
️✔️latest
️✔️3.10
️✔️3.9
️✔️rdbms
️✔️latest
️✔️3.10
️✔️3.9
️✔️redis
️✔️latest
️✔️3.10
️✔️3.9
️✔️relay
️✔️latest
️✔️3.10
️✔️3.9
️✔️resource
️✔️2018-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️role
️✔️latest
️✔️3.10
️✔️3.9
️✔️search
️✔️latest
️✔️3.10
️✔️3.9
️✔️security
️✔️latest
️✔️3.10
️✔️3.9
️✔️servicebus
️✔️latest
️✔️3.10
️✔️3.9
️✔️serviceconnector
️✔️latest
️✔️3.10
️✔️3.9
️✔️servicefabric
️✔️latest
️✔️3.10
️✔️3.9
️✔️signalr
️✔️latest
️✔️3.10
️✔️3.9
️✔️sql
️✔️latest
️✔️3.10
️✔️3.9
️✔️sqlvm
️✔️latest
️✔️3.10
️✔️3.9
️✔️storage
️✔️2018-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️synapse
️✔️latest
️✔️3.10
️✔️3.9
️✔️telemetry
️✔️2018-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9
️✔️util
️✔️latest
️✔️3.10
️✔️3.9
️✔️vm
️✔️2018-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2019-03-01-hybrid
️✔️3.10
️✔️3.9
️✔️2020-09-01-hybrid
️✔️3.10
️✔️3.9
️✔️latest
️✔️3.10
️✔️3.9

@azure-client-tools-bot-prd
Copy link

azure-client-tools-bot-prd bot commented Jul 6, 2023

️✔️AzureCLI-BreakingChangeTest
️✔️Non Breaking Changes

@ghost ghost added Auto-Assign Auto assign by bot CI CI labels Jul 6, 2023
@yonzhan
Copy link
Collaborator

yonzhan commented Jul 6, 2023

CI

@ghost ghost requested review from yonzhan and jsntcy July 6, 2023 02:19
@ghost ghost assigned wangzelin007 Jul 6, 2023
@wangzelin007 wangzelin007 changed the title {CI} Test the cloud module last to avoid affecting the results of other tests. {CI} Put the cloud module at the end of the serial execution to avoid affecting the results of other tests. Jul 6, 2023
@wangzelin007 wangzelin007 changed the title {CI} Put the cloud module at the end of the serial execution to avoid affecting the results of other tests. {CI} Put the cloud module at the end of the serial execution to avoid affecting other tests. Jul 6, 2023
kairu-ms
kairu-ms previously approved these changes Jul 6, 2023
@jsntcy
Copy link
Member

jsntcy commented Jul 6, 2023

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 3 pipeline(s).

@jiasli
Copy link
Member

jiasli commented Jul 6, 2023

The error is caused by performing tenant discovery on mock tenant 00000000-0000-0000-0000-000000000000 by calling https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/v2.0/.well-known/openid-configuration

MOCKED_TENANT_ID = '00000000-0000-0000-0000-000000000000'

My gut feeling tells me this may not be caused by cloud command module, but I need to know where the error is raised. However, it seems the full call stack has been redacted from the CI output by the current testing framework:

https://dev.azure.com/azclitools/public/_build/results?buildId=69514&view=logs&j=1e2e30e3-4bd9-5efc-001c-b6595870bcf5&t=6acccfdf-22ef-5935-fc1e-7a7302be73cb&l=1920

"Content": "|Type|Test Case|Error Message|Line|\n|---|---|---|---|\n|Failed|test_get_docker_credentials|E   ValueError: OIDC Discovery failed on https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/v2.0/.well-known/openid-configuration. HTTP status: 400, Error: {\"error\":\"invalid_tenant\",\"error_description\":\"AADSTS900021: Requested tenant identifier '00000000-0000-0000-0000-000000000000' is not valid. Tenant identifiers may not be an empty GUID.\\r\\nTrace ID: 4f691ff2-1fd0-4261-ab70-e1bf37b54600\\r\\nCorrelation ID: 5ee5b4ef-52f7-48d6-8e24-351d067c7bac\\r\\nTimestamp: 2023-07-05 08:16:30Z\",\"error_codes\":[900021],\"timestamp\":\"2023-07-05 08:16:30Z\",\"trace_id\":\"4f691ff2-1fd0-4261-ab70-e1bf37b54600\",\"correlation_id\":\"5ee5b4ef-52f7-48d6-8e24-351d067c7bac\"}<br><br>During handling of the above exception, another exception occurred:<br>E   ValueError: Unable to get authority configuration for https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000. Authority would typically be in a format of https://login.microsoftonline.com/your_tenant Also please double check your tenant name or GUID is correct.|azure/cli/command_modules/acr/tests/latest/test_acr_commands_mock.py:1141|\n"

Can we get the full call stack back?

@wangzelin007
Copy link
Member Author

The error is caused by performing tenant discovery on mock tenant 00000000-0000-0000-0000-000000000000 by calling https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/v2.0/.well-known/openid-configuration

MOCKED_TENANT_ID = '00000000-0000-0000-0000-000000000000'

My gut feeling tells me this may not be caused by cloud command module, but I need to know where the error is raised. However, it seems the full call stack has been redacted from the CI output by the current testing framework:

https://dev.azure.com/azclitools/public/_build/results?buildId=69514&view=logs&j=1e2e30e3-4bd9-5efc-001c-b6595870bcf5&t=6acccfdf-22ef-5935-fc1e-7a7302be73cb&l=1920

"Content": "|Type|Test Case|Error Message|Line|\n|---|---|---|---|\n|Failed|test_get_docker_credentials|E   ValueError: OIDC Discovery failed on https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/v2.0/.well-known/openid-configuration. HTTP status: 400, Error: {\"error\":\"invalid_tenant\",\"error_description\":\"AADSTS900021: Requested tenant identifier '00000000-0000-0000-0000-000000000000' is not valid. Tenant identifiers may not be an empty GUID.\\r\\nTrace ID: 4f691ff2-1fd0-4261-ab70-e1bf37b54600\\r\\nCorrelation ID: 5ee5b4ef-52f7-48d6-8e24-351d067c7bac\\r\\nTimestamp: 2023-07-05 08:16:30Z\",\"error_codes\":[900021],\"timestamp\":\"2023-07-05 08:16:30Z\",\"trace_id\":\"4f691ff2-1fd0-4261-ab70-e1bf37b54600\",\"correlation_id\":\"5ee5b4ef-52f7-48d6-8e24-351d067c7bac\"}<br><br>During handling of the above exception, another exception occurred:<br>E   ValueError: Unable to get authority configuration for https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000. Authority would typically be in a format of https://login.microsoftonline.com/your_tenant Also please double check your tenant name or GUID is correct.|azure/cli/command_modules/acr/tests/latest/test_acr_commands_mock.py:1141|\n"

Can we get the full call stack back?

I test in this PR, If don't test it with cloud module, ci won't report an error.
Full call stack

@wangzelin007 wangzelin007 merged commit 0694c11 into Azure:dev Jul 10, 2023
53 of 54 checks passed
@jiasli jiasli mentioned this pull request Jul 28, 2023
3 tasks
avgale pushed a commit to avgale/azure-cli that referenced this pull request Aug 24, 2023
… affecting other tests. (Azure#26824)

* Update automation_full_test.py

* Update automation_full_test.py

* Update automation_full_test.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Auto-Assign Auto assign by bot CI CI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants