-
Notifications
You must be signed in to change notification settings - Fork 231
Caching Proxy: Features
When you request the interface of a remote service via dependency injection and the interface contains cached methods, instead of directly getting an instance of the Service Proxy you will get it wrapped in a Caching Proxy. This means any call to the interface will first be handled by the Caching Proxy, which will provide a cached value if available, and if it's not it will call the Service Proxy and cache the result.
This functionality is provided when loading ServiceProxyModule into Ninject, which is loaded by MicrodotModule which in turn is loaded by MicrodotServiceHost and MicrodotOrleansServiceHost base-classes which you extended into your own hosting classes.
The idea behind transparent proxies is that they add a caching layer without requiring code changes. That means they are responsible for generating the cache key using only the intercepted method call. The cache key must be unique for any given method and arguments, so that a change the method or in any argument produces a different cache key.
The Caching Proxy generates the cache key in the following pattern:
$"{TypeName}.{MethodName}({string.Join(",", ParameterTypes)}#{GetArgumentHash(args)})Where GetArgumentHash creates converts all the arguments in an array to JSON, calculates the SHA1 hash of it and converts the hash to a Base64 string. So for example, a call to the following method:
void Task SaveUser(string name, int age);With the arguments ["Alice", 42] will produce the following cache key:
"ICustomerService.SaveUser(string,int)#lfqfcjjeyjOD5bMtKjtJ3wKTEIs="You can define a TTL to evict an item from the cache after a certain period of time has elapsed since it was added. This can prevent stale data from being returned from the cache. If an item is requested from the cache after it has been evicted by TTL, the data source will be used to retrieve an up-to-date version of the item.
The Caching Proxy doesn't support sliding TTLs in its current version, but only absolute TTLs. Sliding TTLs are effectively implemented by background refreshes (see below).
Using configuration, can change the TTL of the cache (default is 6 hours) at either the service level, which affects all cached methods on that service, or at the method level, which affect only a specific method group (all of its overloads). For more information about how to change the caching TTLs in the configurations, see Service Discovery: Configuration.
Traditional TTLs have certain downsides. One of them is "latency spiked". Since there is a very large performance difference between calls that hit the cache and those that miss, when a cache item is invalidated, it immediately causes a spike in latency and a drop in throughput, because all requests that were served quickly from the cache now have to wait for a remote request to complete. Also, invalidation of a cache item also hurts the fault tolerance described since if the item is not available in the cache, any failure of a remote service will be felt immediately.
To avoid those issues, the Caching Proxy implements a soft invalidation mechanism, where an item is only marked as stale rather than actually being evicted. An item is marked as stale after a certain period of time, called Refresh Time, since it's been in the cache, typically much shorter than the TTL. The default is 1 minute, but can be changed using configurations (see Service Discovery: Configuration).
When an item is requested from the cache but only a stale item is available, it is returned to the caller but a background refresh process is started that attempts to get the latest version of the item and replace the stale one that's in the cache with the freshly retrieve version. The background process contacts the data source, and if an item is returned, it puts it in the cache, overwriting the old item. It also resets the TTL and Refresh Time, which has the effect of a sliding TTL. If for any reason the data source doesn't return an item (usually when an exception was thrown), the stale item will remain in the cache and will continue to be served to any callers. An waiting time called Failed Refresh Delay (also configurable, default to 1 second) will be enacted before additional attempts are made to to retrieve the value from the data source, so that it's not "hammered". There is no maximum number of retries, attempts to refresh the data will continue until the TTL elapses and the item is evicted from the cache.
This soft invalidation prevents sudden latency and throughputs changes when the TTL expires (which can be much shorter if soft invalidation wasn't available), and allows the cache to protect against unavailability of a service due to longer TTLS. If using the default TTL of 6 hours and Refresh Time of 1 minutes, data in the cache would be kept up-to-date up to around 1 minute and yet allow for a service to be down for up to 6 hours before the cache no longer has even stale data to serve. This is a good balance of keeping the data fresh and available.
You want to set the Refresh Time to a reasonable amount of time a data can be out of date but still useful. The shorted the time period, the bigger the load on the system and the lower the benefit of the cache.
You want to set the TTL to the maximum staleness of data you're willing to accept, which beyond that time period you'd rather requests start failing than use data so stale. Remember that the TTL is your safety net in case a service fails, at least some requests will be served from the cache. Beyond the TTL you will have 100% failure rates if the service is still down. Configure it according to your SLA and the amount of time it takes you to detect and recover from a service failure.