Add stability test case to test issue #4608

[joviegas]

Motivation and Context

Test #4608

Root cause of #4608

The issue was fixed with revert #4621 and actual fix #4620
Following was the root cause

1. This issue is specific when we have S3 API calls with and without checksum back to back.
2. The Race condition occurs in This code

       return headerChecksumSpecs != null &&
              headerChecksumSpecs.algorithm() != null &&
              !HttpChecksumUtils.isHttpChecksumPresent(interceptorContext.httpRequest(), headerChecksumSpecs) &&
              HttpChecksumUtils.isUnsignedPayload(
                  context.executionAttributes().getAttribute(SIGNING_METHOD), interceptorContext.httpRequest().protocol(),
                  isContentStreaming) &&
              !headerChecksumSpecs.isRequestStreaming();

3. ChecksumThread Has a checksum because Its CHECKSUM_SPECS was updated with non null values here thus its headerChecksumSpecs !=null passes and it goes to HttpChecksumUtils.isHttpChecksumPresent(interceptorContext.httpRequest(), headerChecksumSpecs)
4. Now when a Non Checksum API in another NonChecksumThread acccess the RESOLVED_CHECKSUM_SPECS its header and other attributes are reset to null because no checksum for that API
5. When the API in ChecksumThread resumes , its in call where it does DefaultSdkHttpFullRequest.firstMatchingHeader where it expects the string to be nonNull for a String compare to find the header in a Tree Map, thus ends up with NPE

Modifications

• Added a test case in stability test to send Request with and without checksum back to back

Testing

• Made sure that test is valid by checking on release which had the issue
• git checkout d9cd9d3 and tested issue is recreated with the stabilty test added

License

• I confirm that this pull request can be released under the Apache 2 license

[cenedhryn]

There are dependency issues with module stability-tests. Did you import something from a different module that's not in the pom?

[zoewangg]

There's a chance that all concurrent requests are using the same checksum algorithm, can we pick a checksum algorithm based on index, like index % 4 and each one correlates to one of the supported checksum algorithms? This way, we can remove thenRunAsync

[joviegas]

Done

[joviegas]

There are dependency issues with module stability-tests. Did you import something from a different module that's not in the pom?

Yeap , it was caused by StaticCredentialProvider

[cenedhryn]

Just to confirm - the test fails on the code that has the bug, and it passes using master?

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[joviegas]

Just to confirm - the test fails on the code that has the bug, and it passes using master?

Yeap , (mentioned in the the testing section). Done stability test in PR checks