-
Notifications
You must be signed in to change notification settings - Fork 164
Java, Android Storage Client Library Date Bug Resolution
A bug has been found that affects users of the Java Storage Client Library (version 1.3.1 and below) and the Azure Storage Client Library for Android (preview release versions 0.3.1 and below). This bug occurs only when using the Azure Table Service via TableEntity objects that contain a Date in custom fields within the entity. It does not affect partition keys or row keys containing dates.
The Java Date class (java.util.Date) is used to represent time values in the Storage Client Library. This class has only millisecond precision, whereas .NET DateTime and the Azure Storage Service have precision to the ticks level (where a tick represents 100 nanoseconds). These differences lead to issues when converting between date formats manifesting itself in two separate ways:
This case occurs when using only the Java Storage Client Library or Android Storage Client Library to interact with Date properties on Table Entities.
When writing a date property’s value to the service (eg 2014-12-16T20:42:30.123), this value is incorrectly padded with leading zeros instead of trailing zeros (eg 2014-12-16T20:42:30.0000123Z). This is then returned the same way by the service on a retrieve. In the common case, we then make the same error in reverse, parsing this into a Java Date as intended (eg 2014-12-16T20:42:30.123), which appears to be correct to users.
However, if that Date’s millisecond value has a trailing zero (eg 2014-12-16T20:42:30.120), this trailing zero will be stripped when returned to the library on retrieve (eg 2014-12-16T20:42:30.000012Z) before the SDK parses it incorrectly as above. Thus, the original Java Date does not round trip in this trailing zero case (eg 2014-12-16T20:42:30.120 becomes 2014-12-16T20:42:30.012, changing from 120 ms to 12 ms). The maximum difference between the intended and the actual value in this case is 891ms.
This case occurs when using some mix of the Java or Android Storage Client Library and another Azure Storage Client Library or REST calls to interact with Date properties on Table Entities.
When writing a date property’s value to the service via the Java or Android Library (eg 2014-12-16T20:42:30.123), this value is incorrectly padded with leading zeros instead of trailing zeros (eg 2014-12-16T20:42:30.0000123Z). If this value is retrieved by another Storage Client Library (for instance, .NET), it will be parsed according to the REST specification, treating the final three digits of the value not as milliseconds, but rather, ticks.
In the opposite case, when writing a date property’s value to the service via another Client Library or REST (eg 2014-12-16T20:42:30.1234567), the value is stored as intended. If this value is retrieved by the Java or Android Storage Client Libraries, it will be incorrectly parsed as above, interpreting the number of ten-millionths of a second instead as a number of milliseconds, causing millisecond ‘overflow’ of sorts. The maximum difference between the intended and the actual value is 9999000 ms.
Please note that while the Date values at the millisecond level must inherently lose precision due to the limitations of the language, we do store the underlying strings representing these values returned by the Azure Storage Service. If exact values are needed, we recommend using these strings instead of the converted Date value.
We have updated version 2.0.0 of the Java Storage Client Library, and version 0.4.0 of the Android Storage Client Library to:
-
Correct the insertion/merge/replace (write) behavior so we correctly format any date values and send them to the Storage service correctly (eg 2014-12-16T20:42:30.251 is formatted and stored as 2014-12-16T20:42:30.2510000Z).
-
Fix the trailing zero issue so that any missing zeros are inserted in the correct place (eg 2014-12-16T20:42:30.750 is formatted and stored as 2014-12-16T20:42:30.7500000Z, retrieved as 2014-12-16T20:42:30.75Z and formatted as 2014-12-16T20:42:30.750).
We have also added a flag to TableRequestOptions that specifies whether the customer is both using only the Java or Android Storage Client Libraries to interact with storage, and has data written with older versions of the library – and hence will want to use a Date backward compatibility mode. By default, this would be set to indicate a cross-platform use case, with backward compatibility off. New users of the library, in any scenario, should also use this default behavior. The flag can then be set on CloudTableClient.defaultRequestOptions for all requests, or on a per-request basis by setting it on the request options parameter (See Sample Usage below). Based on the setting, retrieve (read) behavior will be performed on Date properties as follows:
If Table Entities containing Date properties were only written by the Java or Android Storage Client Libraries & Date class, we can determine with certainty whether these values were written before or after this fix was implemented. Because these times can only have millisecond precision, only three of the seven possible subsecond values from the service can be populated upon retrieval – the first three, if it was written correctly (eg 2014-12-16T20:42:30.6840000Z) or the last three if not (eg 2014-12-16T20:42:30.0000684Z). We can then always read these three numbers as the millisecond value of the date, no matter their placement. This would happen entirely under the hood, so customers would get the correct date upon retrieval (eg 2014-12-16T20:42:30.684) no matter if it was written correctly or incorrectly. See the Illustrative Examples section for more details.
In the mixed library use case, we have no way to determine whether data was written to the service by an earlier Java or Android Client incorrectly or by another client correctly (eg 2014-12-16T20:42:30.0000684Z could be a correct .NET DateTime or a Java Date written incorrectly). Therefore, we cannot act intelligently on that existing data when we read. However, the Java Date class can never have any precision past the millisecond level, so truncate any values returned by the service that are more precise. This loss of precision will be minor, and no worse than what already exists in the library with the bug today (eg 2014-12-16T20:42:30.0000684Z will be returned in Java as 2014-12-16T20:42:30.000). See the Illustrative Examples section for more details.
{
// Set the default value for all requests from a CloudTableClient
client.getDefaultRequestOptions().setDateBackwardCompatibility(true);
// OR, create a new options object to use for a specific request
TableRequestOptions options = new TableRequestOptions();
options.setDateBackwardCompatibility(false);
// Then, retrieve operations will use the DateBackwardCompatibility value set on the options
// object if it was passed in, otherwise, the default value set on the client object.
TableOperation retrieve = TableOperation.retrieve(partitionKey, rowKey, MyTableEntity.class);
// Uses value from client.defaultRequestOptions
myEntity = myTable.execute(retrieve).getResultAsType();
// Uses value from passed in TableRequestOptions argument
myEntity = myTable.execute(retrieve, options, null /* operationContext */).getResultAsType();
}Example 1: **Java/Android Only Scenario -**Original Java Date 2014-12-07T09:15:12.123 UTC (yyyy-MM-ddTHH:mm:ss.fff)
Customers using solely the Java and Android Storage Client Libraries and the Java Date class to interact with Table Entity Date Property values will see no issue going forward, and correct reading of the incorrect values on the service if they set the DateBackwardCompatibility to true. If they do not, they run the risk of losing millisecond precision of old data.
| Behavior | OriginalJavaDate (yyyy-MM-ddTHH:mm:ss.fff) | Written To Azure Storage (ISO8601 Long String) | Returned by Azure Storage (ISO8601 Long String) | FinalJavaDate Value (yyyy-MM-ddTHH:mm:ss.fff) | Summary |
|---|---|---|---|---|---|
| Insert & Read with Pre-2.0 | 2014-12-07T09:15:12.123 | 2014-12-07T09:15:12.0000123Z | 2014-12-07T09:15:12.0000123Z | 2014-12-07T09:15:12.123 | Incorrect Value Stored on Service, Correct Value returned to Client |
| Insert with Pre-2.0, Read with 2.0+, DateBackCompat = false | 2014-12-07T09:15:12.123 | 2014-12-07T09:15:12.0000123Z | 2014-12-07T09:15:12.0000123Z | 2014-12-07T09:15:12.000 | Incorrect Value Stored on Service, Loss of ms Precision Returned to Client |
| Insert with Pre-2.0, Read with 2.0+, DateBackCompat = true | 2014-12-07T09:15:12.123 | 2014-12-07T09:15:12.0000123Z | 2014-12-07T09:15:12.0000123Z | 2014-12-07T09:15:12.123 | Incorrect Value Stored on Service, Correct Value Returned to Client |
| Insert & Read with v2.0+, DateBackCompat = false | 2014-12-07T09:15:12.123 | 2014-12-07T09:15:12.1230000Z | 2014-12-07T09:15:12.123Z | 2014-12-07T09:15:12.123 | No Issue |
| Insert & Read with v2.0+, DateBackCompat = true | 2014-12-07T09:15:12.123 | 2014-12-07T09:15:12.1230000Z | 2014-12-07T09:15:12.123Z | 2014-12-07T09:15:12.123 | No Issue |
Example 2: **Java/Android Only Scenario -**Original Java Date 2015-01-14T14:53:32.800 UTC (yyyy-MM-ddTHH:mm:ss.fff)
In the trailing zero millisecond case, customers using solely the Java and Android Storage Client Libraries and the Java Date class to interact with Table Entity Date Property values will see no issue going forward, and correct reading of the incorrect values on the service if they set the DateBackwardCompatibility flag to true. If they incorrectly do not, they run the risk of losing millisecond precision of previously stored data.
| Behavior | OriginalJavaDate (yyyy-MM-ddTHH:mm:ss.fff) | Written To Azure Storage (ISO8601 Long String) | Returned by Azure Storage (ISO8601 Long String) | FinalJavaDate Value (yyyy-MM-ddTHH:mm:ss.fff) | Summary |
|---|---|---|---|---|---|
| Insert & Read with Pre-2.0 | 2015-01-14T14:53:32.800 | 2015-01-14T14:53:32.0000800Z | 2015-01-14T14:53:32.00008Z | 2015-01-14T14:53:32.008 | Incorrect Value Stored on Service, Incorrect Value Returned to Client |
| Insert with Pre-2.0, Read with 2.0+, DateBackCompat = false | 2015-01-14T14:53:32.800 | 2015-01-14T14:53:32.0000800Z | 2015-01-14T14:53:32.00008Z | 2015-01-14T14:53:32.000 | Incorrect Value Stored on Service, Loss of ms Precision to Client |
| Insert with Pre-2.0, Read with 2.0+, DateBackCompat = true | 2015-01-14T14:53:32.800 | 2015-01-14T14:53:32.0000800Z | 2015-01-14T14:53:32.00008Z | 2015-01-14T14:53:32.800 | Incorrect Value Stored on Service, Correct Value Returned to Client |
| Insert & Read with v2.0+, DateBackCompat = false | 2015-01-14T14:53:32.800 | 2015-01-14T14:53:32.8000000Z | 2015-01-14T14:53:32.8Z | 2015-01-14T14:53:32.800 | No Issue |
| Insert & Read with v2.0+, DateBackCompat = true | 2015-01-14T14:53:32.800 | 2015-01-14T14:53:32.8000000Z | 2015-01-14T14:53:32.8Z | 2015-01-14T14:53:32.800 | No Issue |
Example 3: Cross Platform Scenario - Original .NET Date 2014-11-29T22:55:21.9876543Z (ISO 8601 Long)
In the case where the previously stored value is clearly not written by the Pre-2.0 Java Storage Client Library, customers using another means to insert Table Entity Date Property values and reading them using the Java Storage Client Library will see no issue going forward, except a loss of tick-level precision necessitated by the limitations of the Java Date class.
| Behavior | Original**.NET****Date** (ISO 8061 Long) | Written To Azure Storage (ISO8601 Long String) | Returned by Azure Storage (ISO8601 Long String) | FinalJavaDate Value (yyyy-MM-ddTHH:mm:ss.fff) | Summary |
|---|---|---|---|---|---|
| Insert with other means, Read with Pre-2.0 | 2014-11-29T20:55:21.9876543Z | 2014-11-29T20:55:21.9876543Z | 2014-11-29T20:55:21.9876543Z | 2014-11-29T23:39:57.543 | Correct Value Stored, Incorrect Value Returned |
| Insert with other means, Read with 2.0+, DateBackCompat = false | 2014-11-29T20:55:21.9876543Z | 2014-11-29T20:55:21.9876543Z | 2014-11-29T20:55:21.9876543Z | 2014-11-29T23:55:21.987 | Correct Value Stored, Language Necessitated Loss of Tick Precision |
| Insert with other means, Read with 2.0+, DateBackCompat = true | 2014-11-29T20:55:21.9876543Z | 2014-11-29T20:55:21.9876543Z | 2014-11-29T20:55:21.9876543Z | 2014-11-29T20:55:21.987 | Correct Value Stored, Language Necessitated Loss of Tick Precision |
Example 4: Cross Platform Scenario - Original .NET Date 2015-02-14T03:11:13.0000229Z (ISO 8601 Long)
In the case where the previously stored value could possibly have been written by the Pre-2.0 Java Storage Client Library, customers using another means to insert Table Entity Date Property values and reading them using the Java Storage Client Library will see differing behavior dependent on how they set the DateBackwardCompatibility flag. If they incorrectly set it to true, they will see incorrect values returned as before. If they set it to false, they will see no issue going forward, except a loss of tick-level precision necessitated by the limitations of the Java Date class.
| Behavior | Original**.NET****Date** (ISO 8601 Long) | Written To Azure Storage (ISO8601 Long String) | Returned by Azure Storage (ISO8601 Long String) | FinalJavaDate Value (yyyy-MM-ddTHH:mm:ss.fff) | Summary |
|---|---|---|---|---|---|
| Insert with other means, Read with Pre-2.0 | 2015-02-14T03:11:13.0000229Z | 2015-02-14T03:11:13.0000229Z | 2015-02-14T03:11:13.0000229Z | 2015-02-14T03:11:13.229 | Correct Value Stored, Incorrect Value Returned |
| Insert with other means, Read with 2.0+, DateBackCompat**= false** | 2015-02-14T03:11:13.0000229Z | 2015-02-14T03:11:13.0000229Z | 2015-02-14T03:11:13.0000229Z | 2015-02-14T03:11:13.000 | Correct Value Stored, Language Necessitated Loss of Tick Precision |
| Insert with other means, Read with 2.0+, DateBackCompat**= true** | 2015-02-14T03:11:13.0000229Z | 2015-02-14T03:11:13.0000229Z | 2015-02-14T03:11:13.0000229Z | 2015-02-14T03:11:13.229 | Correct Value Stored, Incorrect Value Returned |
Example 5: Cross Platform Scenario – Java Date 2014-12-13T18:51:29.504 UTC (yyyy-MM-ddTHH:mm:ss.fff)
In the case where the stored value was written by the Java or Android Storage Client Libraries, customers using another means to read Table Entity Date Property values will read old values incorrectly, but will see no issues going forward.
| Behavior | OriginalJavaDate (yyyy-MM-ddTHH:mm:ss.fff) | Written To Azure Storage (ISO8601 Long String) | Returned by Azure Storage (ISO8601 Long String) | Final**.NET****Date Value** (ISO 8601 Long) | Summary |
|---|---|---|---|---|---|
| Written with Pre-2.0, Read with other means | 2014-12-13T18:51:29.504 | 2014-12-13T18:51:29.0000504Z | 2014-12-13T18:51:29.0000504Z | 2014-12-13T18:51:29.0000504Z | Incorrect value stored, Incorrect Value Returned |
| Written with 2.0+, Read with other means | 2014-12-13T18:51:29.504 | 2014-12-13T18:51:29.5040000Z | 2014-12-13T18:51:29.5040000Z | 2014-12-13T18:51:29.5040000Z | No Issue |
We recommend an immediate update to version 2.0.0 of the Java Storage Client Library and version 0.4.0 of the Android Storage Client Library in order to minimize the impact of this bug going forward. We also recommend being cautious when setting the TableRequestOptions DateBackwardCompatibility to the correct value – changing it to true if and only if you use the Java or Android Storage Client Libraries and the Date class only to store Date Property Values on Table Entities --- before making further Table service requests. Please note that the default for this setting is false. Again, while the Date values at the millisecond level must inherently lose precision due to the limitations of the language, we do store the underlying strings representing these values returned by the Azure Storage Service. If exact values are needed, we recommend using these strings (by calling EntityProperty.getVaueAsString()) instead of the converted Date value.
We'd like to thank GitHub users mooso and MaltAlex for helping to identify this issue.
Peter Marino & The Azure Storage Team