Getting occasional Crc32MismatchException using DynamoDbClient

[dev-k13]

Describe the bug

We are getting Crc32MismatchException when doing a query to DynamoDB from our application. Happened only on few occasions but most of the time it is working fine.

On the two instances that this exception occurred, application was only retrieving 14596 bytes and 224264 bytes. For the instance that was reading 14596 bytes, read was successful after performing a retry.

Expected Behavior

Query from DynamoDB without getting checksum validation error.

Current Behavior

software.amazon.awssdk.core.exception.Crc32MismatchException: Expected 2932621229 as the Crc32 checksum but the actual calculated checksum was 750835520
	at software.amazon.awssdk.core.exception.Crc32MismatchException$BuilderImpl.build(Crc32MismatchException.java:97)
	at software.amazon.awssdk.core.internal.util.Crc32ChecksumValidatingInputStream.validateChecksum(Crc32ChecksumValidatingInputStream.java:62)
	at software.amazon.awssdk.core.internal.util.Crc32ChecksumValidatingInputStream.close(Crc32ChecksumValidatingInputStream.java:50)
	at java.base/java.io.FilterInputStream.close(FilterInputStream.java:179)
	at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeRunnable$5(FunctionalUtils.java:124)
	at software.amazon.awssdk.utils.FunctionalUtils.invokeSafely(FunctionalUtils.java:140)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonResponseHandler.lambda$handle$2(JsonResponseHandler.java:87)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonResponseHandler.handle(JsonResponseHandler.java:87)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.JsonResponseHandler.handle(JsonResponseHandler.java:36)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.AwsJsonResponseHandler.handle(AwsJsonResponseHandler.java:43)
	at software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.lambda$handle$0(MetricCollectingHttpResponseHandler.java:52)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:63)
	at software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.handle(MetricCollectingHttpResponseHandler.java:52)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:95)
	at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$resultTransformationResponseHandler$8(BaseClientHandler.java:283)
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleSuccessResponse(CombinedResponseHandler.java:99)
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:75)
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:60)
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:41)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipe

Reproduction Steps

DynamoDbClient dynamoDbClient = DynamoDbClient.builder().httpClientBuilder(ApacheHttpClient.builder()).build();

QueryRequest queryRequest = QueryRequest.builder()
			                .tableName(tableName)
			                .keyConditionExpression(keyConditionExpression)
			                .expressionAttributeNames(expressionAttributeNames)
			                .expressionAttributeValues(expressionAttributeValues)
			                .exclusiveStartKey(lastKeyEvaluated)
			                .build();

QueryResponse response = dynamoDbClient.query(queryRequest);

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.19.17

JDK version used

17

Operating System and version

Amazon Linux

[debora-ito]

@dev-k13 Good timing, a fix for this went out last week:

https://github.com/aws/aws-sdk-java-v2/blob/master/CHANGELOG.md#aws-sdk-for-java-v2-1

Please upgrade to Java SDK version >= 2.25.55, and let us know if you still experience the error afterwards.

[dev-k13]

@dev-k13 Good timing, a fix for this went out last week:

https://github.com/aws/aws-sdk-java-v2/blob/master/CHANGELOG.md#aws-sdk-for-java-v2-1

Please upgrade to Java SDK version >= 2.25.55, and let us know if you still experience the error afterwards.

@debora-ito thanks for the prompt update. We will plan for upgrade. However, just wanted to clarify, the changelog for the bugfix states:

Prevent calculating crc32 checksum on an incomplete response, due to error

Does this mean that prior the 2.25.55 fix, checksum validation is being triggered when an error occurs during read and upgrading to 2.25.55 will possibly show the underlying error causing the incomplete response?

[debora-ito]

and upgrading to 2.25.55 will possibly show the underlying error causing the incomplete response?

Yes, here's the PR for reference: #5224

In one case that we analyzed, "I/O error: Read timed out" was the error that triggered the checksum calculation, and the "Read timed out" message was logged at DEBUG level.

[dev-k13]

and upgrading to 2.25.55 will possibly show the underlying error causing the incomplete response?

Yes, here's the PR for reference: #5224

In one case that we analyzed, "I/O error: Read timed out" was the error that triggered the checksum calculation, and the "Read timed out" message was logged at DEBUG level.

Okay, we will upgrade and see how it goes. Thanks, @debora-ito!

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.