Epic: IDistributedCache updates in .NET 9

[adityamandaleeka]

Update:

HybridCache has relocated to dotnet/extensions:dev; it does not ship in .NET 9 RC1, as a few missing and necessary features are still in development; however, we expect to ship either alongside or very-shortly-after .NET 9! ("extensions" has a different release train, that allows additional changes beyond the limit usually reserved for in-box packages; HybridCache has always been described as out-of-box - i.e. a NuGet package - so: there is no reason for us to limit ourselves by the runtime restrictions)

---

Status: feedback eagerly sought

• API proposal part 1, core API: Hybrid Cache API proposal #54647 and Add HybridCache public API #55084
• Initial cut of proposal part 1: Implement minimal implementation of HybridCache #55147 and fixes: HybridCache - minor post-PR fixes #55251
• API proposal part 2, tags and invalidation: HybridCache - tags and invalidation #55308
• By nomenclature: HybridCache - rename RemoveKeyAsync and RemoveTagAsync to "By" #55332

Tl;Dr

• add a new HybridCache API (and supporting pieces) to support more convenient and efficient distributed cache usage
• support read-through caching with lambda callbacks
• support flexible serialization
• support stampede protection
• support L1/L2 cache scenarios
• build on top of IDistributedCache so that all existing cache backends work without change (although they could optionally add support for new features)
• support comparable expiration concepts to IDistributedCache

Problem statement

The distributed cache in asp.net (i.e. IDistributedCache) is not particularly developed; it is inconvenient to use, lacks many desirable features, and is inefficient. We would like this API to be a "no-brainer", easy to get right feature, making it desirable to use - giving better performance, and a better experience with the framework.

Typical usage is shown here; being explicit about the problems:

Inconvenient usage

The usage right now is extremely manual; you need to:

• attempt to read a stored value (as byte[])
• check that value for null ("no value")
  • if not null:
    • fetch the value
    • serialize it
    • store the value
  • return the value

This is a lot of verbose boilerplate, and while it can be abstracted inside projects using utility methods (often extension methods), the vanilla experience is very poor.

Inefficiencies

The existing API is solely based on byte[]; the demand for right-sized arrays means no pooled buffers can be used. This broadly works for in-process memory-based caches, since the same byte[] can be returned repeatedly (although this implicitly assumes the code doesn't mutate the data in the byte[]), but for out-of-process caches this is extremely inefficient, requiring constant allocation.

Missing features

The existing API is extremely limited; the concrete and implementation-specific IDistributedCache implementation is handed directly to callers, which means there is no shared code reuse to help provide these features in a central way. In particular, there is no mechanism for helping with "stampede" scenarios - i.e. multiple concurrent requests for the same non-cached value, causing concurrent backend load for the same data, whether due to a cold-start empty cache, or key invalidation. There are multiple best-practice approaches that can mitigate this scenario, which we do not currently employ.

Likewise, we currently assume an in-process or out-of-process cache implementation, but caching almost always benefits from multi-tier storage, with a limited in-process (L1) cache supplemented by a separate (usually larger) out-of-process (L2) cache; this gives the "best of both" world, where the majority of fetches are served efficiently from L1, but cold-start and less-frequently-accessed data still doesn't hammer the underlying backend, thanks to L2. Multi-tier caching can sometimes additionally exploit cache-invalidation support from the L2 implementation, to provide prompt L1 invalidation as required.

This epic proposes changes to fill these gaps

Current code layout

At the moment the code is split over multiple components, in the main runtime, asp.net, and external packages (only key APIs shown):

• Microsoft.Extensions.Caching.Abstractions

  • src: https://github.com/dotnet/runtime/tree/main/src/libraries/Microsoft.Extensions.Caching.Abstractions
  • pkg: https://www.nuget.org/packages/Microsoft.Extensions.Caching.Abstractions/
  • adds IDistributedCache
  • adds DistributedCacheEntryOptions

• Microsoft.Extensions.Caching.Memory

  • src: https://github.com/dotnet/runtime/tree/main/src/libraries/Microsoft.Extensions.Caching.Memory
  • pkg: https://www.nuget.org/packages/Microsoft.Extensions.Caching.Memory/
  • adds AddDistributedMemoryCache
  • adds MemoryDistributedCache : IDistributedCache

• Microsoft.Extensions.Caching.StackExchangeRedis

  • src: https://github.com/dotnet/aspnetcore/tree/main/src/Caching/StackExchangeRedis
  • pkg: https://www.nuget.org/packages/Microsoft.Extensions.Caching.StackExchangeRedis
  • adds AddStackExchangeRedisCache

• Microsoft.Extensions.Caching.SqlServer

  • src: https://github.com/dotnet/aspnetcore/tree/main/src/Caching/SqlServer
  • pkg: https://www.nuget.org/packages/Microsoft.Extensions.Caching.SqlServer
  • adds AddDistributedSqlServerCache

• Microsoft.Extensions.Caching.Cosmos

  • src: https://github.com/Azure/Microsoft.Extensions.Caching.Cosmos
  • pkg: https://www.nuget.org/packages/Microsoft.Extensions.Caching.Cosmos
  • adds AddCosmosCache

• Alachisoft.NCache.OpenSource.SDK

  • src: https://www.alachisoft.com/ncache/
  • pkg: https://www.nuget.org/packages/Alachisoft.NCache.OpenSource.SDK/
  • adds AddNCacheDistributedCache

• AWS

  • https://github.com/awslabs/aws-dotnet-distributed-cache-provider

This list is not exhaustive - other 3rd-party and private implementations of IDistributedCache exist, and we should avoid breaking the world.

Proposal

The key proposal here is to add a new caching abstraction that is more focused, HybridCache, in Microsoft.Extensions.Caching.Abstractions; this API is designed to act more as a read-through cache, building on top[ of the existing IDistributedCache implementation, providing all the implementation details required for a rich experience. Additionally, while simple defaults are provided for the serializer, it is an explicit aim to make such concerns fully configurable, allowing for json, protobuf, xml, etc serialization as appropriate to the consumer.

namespace Microsoft.Extensions.Caching.Distributed;

public abstract class HybridCache // default concrete impl provided by service registration
{
    protected HybridCache() { }

    // read-thru usage
    public abstract ValueTask<T> GetOrCreateAsync<TState, T>(string key, TState state, Func<TState, CancellationToken, ValueTask<T>> callback, HybridCacheEntryOptions? options = null, ReadOnlyMemory<string> tags = default, CancellationToken cancellationToken = default);
    public virtual ValueTask<T> GetOrCreateAsync<T>(string key, Func<CancellationToken, ValueTask<T>> callback,
    HybridCacheEntryOptions? options = null, ReadOnlyMemory<string> tags = default, CancellationToken cancellationToken = default)
    { /* shared default implementation uses TState/T impl */ }

    // manual usage
    public abstract ValueTask<(bool Exists, T Value)> GetAsync<T>(string key, HybridCacheEntryOptions? options = null, CancellationToken cancellationToken = default);
    public abstract ValueTask SetAsync<T>(string key, T value, HybridCacheEntryOptions? options = null, ReadOnlyMemory<string> tags = default, CancellationToken cancellationToken = default);

    // key invalidation
    public abstract ValueTask RemoveKeyAsync(string key, CancellationToken cancellationToken = default);
    public virtual ValueTask RemoveKeysAsync(ReadOnlyMemory<string> keys, CancellationToken cancellationToken = default)
    { /* shared default implementation uses RemoveKeyAsync */ }

    // tag invalidation
    public virtual ValueTask RemoveTagAsync(string tag, CancellationToken cancellationToken = default)
    { /* shared default implementation uses RemoveTagsAsync */ }
    public virtual ValueTask RemoveTagsAsync(ReadOnlyMemory<string> tags, CancellationToken cancellationToken = default) => default;
}

Notes:

• the intent is that instead of requesting IDistributedCache, consumers might use HybridCache; to enable this, the consumer must additionally perform a services.AddHybridCache(...); step during registration
• the naming of GetOrCreateAsync<T> is for parity with MemoryCache.GetOrCreateAsync<T>
• RemoveAsync and RefreshAsync mirror the similarIDistributedCache methods
• it is expected that the callback (when invoked) will return a non-null value; consistent with MemoryCache et-al, null is not a supported value, and an appropriate runtime error will be raised

Usage of this API is then via a read-through approach using lambda; the simplest (but slightly less efficient) approach would be simply:

// HybridCache injected via DI
var data = await cache.GetOrCreateAsync(key, _ => /* some backend read */, [expiration etc], [cancellation]);

In this simple usage, it is anticipated that "captured variables" etc are used to convey the additional state required, as is common for lambda scenarios. A second "stateful" API is provided for more advanced scenarios where the caller wishes to trade convenience for efficiency; this usage is slightly more verbose but will be immediately familiar to the users who would want this feature:

// HybridCache injected via DI
var data = await cache.GetOrCreateAsync(key, (some state here), static (state, _) => /* some backend read */, [expiration etc], [cancellation]);

This has been prototyped and works successfully with type inference etc.

The implementation (see later) deals with all the backend fetch, testing, serialization etc aspects internally.

(in both examples, the "discard" (_) is conveying the CancellationToken for the backend read, and can be used by providing a receiving lambda parameter)

An internal implementation of this API would be registered and injected via a new AddHybridCache API (Microsoft.Extensions.Caching.Abstractions):

namespace Microsoft.Extensions.Caching.Distributed;

public static class HybridCacheServiceExtensions
{
    public static IServiceCollection AddHybridCache(this IServiceCollection services, Action<HybridCacheOptions> setupAction)
    {...}

    public static IServiceCollection AddHybridCache(this IServiceCollection services)
    {...}
}

The internal implementation behind this would receive IDistributedCache for the backend, as it exists currently; this means that the new implementation can use all existing distributed cache backends. By default, AddDistributedMemoryCache is also assumed and applied automatically, but it is intended that this API be effective with arbitrary IDistributedCache backends such as redis, SQL Server, etc. However, to address the issue of byte[] inefficiency, a new entirely optional API is provided and tested for; if the new backend is detected, lower-allocation usage is possible. This follows the pattern used for output-cache in net8:

namespace Microsoft.Extensions.Caching.Distributed;

public interface IBufferDistributedCache : IDistributedCache
{
    ValueTask<CacheGetResult> GetAsync(string key, IBufferWriter<byte> destination, CancellationToken cancellationToken);
    ValueTask SetAsync(string key, ReadOnlySequence<byte> value, DistributedCacheEntryOptions options, CancellationToken cancellationToken);
}

public readonly struct CacheGetResult
{
    public CacheGetResult(bool exists);
    public CacheGetResult(DateTime expiry);

    public CacheGetResult(TimeSpan expiry);

    public bool Exists { get; }
    public TimeSpan? ExpiryRelative { get; }
    public DateTime? ExpiryAbsolute { get; }
}

(the intent of the usual members here is to convey expiration in the most appropriate way for the backend, relative vs absolute, although only one can be specified; the internals are an implementation detail, likely to use overlapped 8-bytes for the DateTime/TimeSpan, with a discriminator)

In the event that the backend cache implementation does not yet implement this API, the byte[] API is used instead, which is exactly the status-quo, so: no harm. The purpose of CacheGetResult is to allow the backend to convey backend expiration information, relevant for L1+L2 scenarios (design note: async precludes out TimeSpan?; tuple-type result would be simpler, but is hard to tweak later). The expiry is entirely optional and some backends may not be able to convey it, and we need to handle it lacking when IBufferDistributedCache is not supported - in either event, the inbound expiration relative to now will be assumed for L1 - not ideal, but the best we have.

Serialization

For serialization, a new API is proposed, designed to be trivially implemented by most serializers - again, preferring modern buffer APIs:

namespace Microsoft.Extensions.Caching.Distributed;

public interface IHybridCacheSerializer<T>
{
    T Deserialize(ReadOnlySequence<byte> source);
    void Serialize(T value, IBufferWriter<byte> target);
}

Inbuilt handlers would be provided for string and byte[] (and possibly BinaryData if references allow); an extensible serialization configuration API supports other types - by default, an inbuilt object serializer using System.Text.Json would be assumed, but it is intended that alternative serializers can be provided globally or per-type. This is likely to be for more efficient bandwidth scenarios, such as protobuf (Google.Protobuf or protobuf-net) etc, but could also be to help match pre-existing serialization choices. While manually registering a specific IHybridCacheSerializer<Foo> should work, it is also intended to generalize the problem of serializer selection, via an ordered set of serializer factories, specifically by registering some number of:

namespace Microsoft.Extensions.Caching.Distributed;

public interface IHybridCacheSerializerFactory
{
    bool TryCreateSerializer<T>([NotNullWhen(true)] out IHybridCacheSerializer<T>? serializer);
}

By default, we will register a specific serializer for string, and a single factory that uses System.Text.Json, however external library implementations are possible, for example:

namespace Microsoft.Extensions.Caching.Distributed;

[SuppressMessage("ApiDesign", "RS0016:Add public types and members to the declared API", Justification = "demo code only")]
public static class ProtobufDistributedCacheServiceExtensions
{
    public static IServiceCollection AddHybridCacheSerializerProtobufNet(this IServiceCollection services)
    {
        ArgumentNullException.ThrowIfNull(services);
        services.AddSingleton<IHybridCacheSerializerFactory, ProtobufNetSerializerFactory>();
        return services;
    }

    private sealed class ProtobufNetSerializerFactory : IHybridCacheSerializerFactory
    {
        public bool TryCreateSerializer<T>([NotNullWhen(true)] out IHybridCacheSerializer<T>? serializer)
        {
            // in real implementation, would use library rules
            if (Attribute.IsDefined(typeof(T), typeof(DataContractAttribute)))
            {
                serializer = new ProtobufNetSerializer<T>();
                return true;
            }
            serializer = null;
            return false;
        }
    }
    internal sealed class ProtobufNetSerializer<T> : IHybridCacheSerializer<T>
    {
        // in real implementation, would use library serializer
        public T Deserialize(ReadOnlySequence<byte> source) => throw new NotImplementedException();

        public void Serialize(T value, IBufferWriter<byte> target) => throw new NotImplementedException();
    }
}

The internal implementation of HybridCache would lookup T as needed, caching locally to prevent constantly using the factory API.

Additional functionality

The internal implementation of HybridCache should also:

• hold the necessary state to serve concurrent requests for the same key from the same incomplete task, similar to the output-cache implementation
• hold the necessary state to support L1/L2 caching
• optionally, support L1 invalidation by a new optional invalidation API

Note that it is this additional state for stampede and L1/L2 scenarios (and the serializer choice, etc) that makes it impractical to provide this feature simply as extension methods on the existing IDistributedCache.

The new invalidation API is anticipated to be something like:

namespace Microsoft.Extensions.Caching.Distributed;

public interface IDistributedCacheInvalidation : IDistributedCache
{
    event Func<string, ValueTask> CacheKeyInvalidated;
}

(the exact shape of this API is still under discussion)

When this is detected, the event would be subscribed to perform L1 cache invalidation from the backend.

Additional things to be explored for HybridCacheOptions:

• options for L1 / L2 caching; perhaps enabled by default if we have IDistributedCacheInvalidation ?
• eager pre-fetch, i.e. "you've asked for X, and the L1 value is still valid, but only just; I'll give you the L1 value, but I'll kick off a fetch against the backend, so there is not a delay when it expires shortly" (disabled by default, due to concerns over lambdas and captured state mutation)
• compression (disabled by default, for simple compatibility with existing data)
• ...?

Additional modules to be enhanced

To validate the feature set, and to provide the richest experience:

• Microsoft.Extensions.Caching.StackExchangeRedis should gain support for IBufferDistributedCache and IDistributedCacheInvalidation - the latter using the "server-assisted client-side caching" feature in Redis
• Microsoft.Extensions.Caching.SqlServer should gain support for IBufferDistributedCache, if this can be gainful re allocatiuons
• guidance should be offered to the Microsoft.Extensions.Caching.Cosmos owners, and if possible: Alachisoft.NCache.OpenSource.SDK

Open issues

• does the approach sound agreeable?
• naming
• where (in terms of packages) does the shared implementation go? in particular, it may need access to System.Text.Json, and possible an L1 implementation ( which could be System.Runtime.Caching, Microsoft.Extensions.Caching.Memory, this new one, or something else) and possibly compression; maybe a new Microsoft.Extensions.Caching.Distributed ? but if so, should it be in-box with .net, or just NuGet? or somewhere else?
• the exact choice of L1 cache (note: this should be an implementation detail; we don't need L1+L2 for MVP)
• how exactly to configure the serializer
• options for eager pre-fetch TTL and enable/disable L1+L2, via TypedDistributedCacheOptions
• should we add tagging support at this juncture?

[aKzenT]

I know it's difficult to change the existing interface, but depending on the other changes planned in this epic, it would be great if we could have distributed cachr methods that were more performance oriented, working with Span instead of byte[].

A common scenario for using the cache is saving serialized objects or large texts encoded as UTF-8. Requiring byte[] usually means copying this data at least once.

Similar issues exist when reading from the cache which also returns a byte[] and thus does not allow for using rented buffers or similar optimizations.

As the cache is often used several times for each request in any non trivial web application (e.g. session store, query cache, response cache), optimizations here would really pay off.

[mgravell]

Yup. Totally understand, @aKzenT , and that's part of the prototype. Will update with more details of the proposed API as it evolves, but the short version here is:

• serialization will be moved inside the cache layer
• pluggable serialization to be based on ReadOnlySequence-byte and IBufferWriter-byte
• new optional backend (store) API to allow buffer usage, but existing byte[] backends wil continue to work (albeit with less efficiency than they could achieve by implementing the new backend API)

[mgravell]

@aKzenT please see updated body

[Tornhoof]

As for L1+L2 caching, you might want to talk to developers of MS FASTER https://github.com/microsoft/FASTER, which has L1+L2 support, while L2 is not strictly out of process, more like out of main memory (disk based or azure based if I remember correctly).
As from my own experience with MS FASTER, it is not necessarily easy to configure properly, but covers a lot of the functionality for L1/L2.

[mgravell]

@Tornhoof aye, FASTER has come up more than a few times; because of the setup etc required, I doubt we can reasonably make that a default implementation; the most pragmatic solution there might be to provide an IDistributedCache (i.e. L2) implementation that is FASTER-based, and leave L1+L2 disabled; that would 100% be something I'd love to see done, but it doesn't need to be critical-path for this epic

[aKzenT]

@mgravell thank you a lot for the update. I'm seeing a lot of things addressed that I've missed in the past, working on multi-server web apps.

For L1+L2 Caching we have been quite happy with https://github.com/ZiggyCreatures/FusionCache in the past which is also built upon IDistributedCache. I remember that there were some features missing from IDistributedCache that made it hard to implement advanced scenarios in that library. So I would like to invite @jodydonetti to this discussion as well as he can probably best comment on these issues.

One thing I remember that was missing was being able to modify the cache entry options (i.e. life time) of a cache entry without going through a Get/Set cycle. Being able to modify the lifetime allows for some advanced scenarios like invalidating a cache entry from the client (e.g. you received a webhook notifying you about changes in data) or reducing the time to allow things like stale results.

Another thing related to cache invalidation, that is not really possible with the current API in an efficient way, is the removal/invalidation of a group of related cache entries. Let's say you have a cache for pages of a CMS system with each page being one entry. The CMS informs you about changes via a web hook, which invalidates the cache for all pages. Directly refreshing all pages might be expensive, so you would rather refresh them individually on demand. So you want to invalidate all page entries in the cache, but there is no way to get the list of entries from the cache, nor is there a good way to delete the entries. Our solution was to built this functionality ourself using a Redis Set that manages the related keys and then iterating through these keys and removing them one by one. But it felt very hacky as you cannot even use the same Redis Connection that the distributed cache uses, as far as I remember.

[mgravell]

@aKzenT re individual cache invalidation: there is a remove API that is meant to serve that function, but it doesn't allow modify of the options; I'd love to understand the need there further

re group cache invalidations: that sounds a lot like the "tag" feature of output-cache, i.e. you associate entries with zero, one or more tags, and then you can invalidate an entire tag, which nukes everything associated; the problem is: that tracking still needs to go somewhere, and it isn't necessarily an inbuilt feature of the cache backend - it was a PITA to implement reasonably on redis without breaking the cluster distribution, for example (ask me how I know!). It also isn't something that can fit in the existing IDistributedCache backend without significant work. Maybe there is some capacity there if we simplified to "zero or one" tags, but even then... I'm not sure that is something we can tackle in this epic, but I'm open to being wrong there!

[mgravell]

Re FusionCache: that isn't one I've seen before, but glancing at the homepage, I see that the fundamental design is pretty similar (some differences, but: very comparable). There is a common theme in these things - indeed, a lot of inspiration (not actual code) in the proposal draws on another implementation of the same that we had when I was at Stack Overflow (in that version, we also had some Roslyn analyzers which complained about inappropriate captured / ambient usage - very neat!). My point: lots of approaches converging around the same API shape.

[aKzenT]

@mgravell we had the same experience implementing our invalidation logic for a redis cluster setup. It's really hard to get right. I would not expect the design to provide a complete solution to this issue, but maybe there is some way that would make it possible for other libraries to support that while building on top of IDistributedCache. Maybe @jodydonetti has an idea how that could work.

As for modifying the options, in the case of FusionCache there is the possibility to allow stale entries, which are treated as expired, but still available in case the backend is not available. For these cases there is a separate timeout of how long you want to allow a result being stale. The TTL that is sent to the underlying IDistributedCache is then the initial TTL plus the additional stale timeout. So when you invalidate an entry, but still want to allow stale results, you cannot simply delete the entry. Instead you would want to update the timeout to being equal to the stale timeout. Hope that makes sense.

[mgravell]

Yep, very familiar with the idea - it is conceptually related to the "eager pre-fetch" mentioned above - with two different TTLs with slightly different meanings

[mgravell]

via twitter:

Yes, it does lack proper API there is too much redundant code we are required to write or maintain a library on our end I used the DistributedCache code snippet provided in one of the .NET blog post by you and it is definitely going to be nice to have these features.

This is referring to this repo, which offered some of the highlights of this proposal as extension methods, but without any of the "meat" that drives the additional functionality.

[aKzenT]

One thing I'm wondering is, if the choice to put the generic type parameter on the interface rather than the methods might be limitting in some cases and would require some classes to have to configure and inject multiple IDistributedCache instances. I'm not sure if that is really a problem, but it would be nice to learn, why you went for that choice, which differs from other implementations that I have seen.

[mgravell]

@aKzenT fair question, and it isn't set in stone, but - reasons:

1. to allow the serializer to be injected, although that might also be possible by taking IServiceProvider
2. to allow the callback signature to be inferred in both the stateless and stateful case, although this might also be possible with a <,> method

Tell you what, I'll branch my branch and try it the other way. Although then I need a new name... dammit!

[mgravell]

@aKzenT ^^^ isn't terrible; will add notes above - we can probably go that way; I probably hadn't accounted for the improvements in generic handling in the last few C# versions (especially with delegates); in particular, I didn't have to change the usage code at all - see L107 for entirety of the usage update

I will benchmark the perf of GetService() as it applies here, but to be honest: in any reasonable L1 cache-hit scenario, I would expect the overall performance to be dominated by the deserialize; and in any L2 or backend scenario, we should anticipate the GetService() to be a rounding error compared to the L2/backend work (if it isn't: why are we caching?)

[danielmarbach]

Would it be a option to use change token for the cache key invalidation instead of the event type proposed currently?

[mgravell]

@danielmarbach in reality, I'm not sure that is feasible in this scenario; the example uses shown for that API seem pretty low-volume and low issuance frequency; file config changes etc, but if we start talking about cache: we're going to need as many change tokens as we have cached entries, and crucially: the backend layer would need to know about them; I'm mentally comparing that to how redis change notifications can work, and to do that efficiently: we don't want to store anything extra, especially at all the backend layers. Given that all we actually want/need here is the string, this seems like going a very long way out of shape, to reuse an API for the sake of it. Happy to keep considering alternatives if I'm missing something, though! Please keep it coming, that's the entire point of this!

[aKzenT]

@danielmarbach I believe it should be fairly easy to implement a GetChangeToken(string key) method as an extension method that subscribes to the event and listens to changes to the specific key. The other way arround is harder. That being said, I'm a fan of ChangeTokens and IIRC, MemoryCache uses change tokens to invalidate entries, so there might be some value to provide such an extension method directly in the framework in addition to the event @mgravell .

[normj]

To add to your list of "Current Code Layout" we have an implementation at AWS for DynamoDB as backend. https://github.com/awslabs/aws-dotnet-distributed-cache-provider

[aKzenT]

@aKzenT ^^^ isn't terrible; will add notes above - we can probably go that way; I probably hadn't accounted for the improvements in generic handling in the last few C# versions (especially with delegates); in particular, I didn't have to change the usage code at all - see L107 for entirety of the usage update

I will benchmark the perf of GetService() as it applies here, but to be honest: in any reasonable L1 cache-hit scenario, I would expect the overall performance to be dominated by the deserialize; and in any L2 or backend scenario, we should anticipate the GetService() to be a rounding error compared to the L2/backend work (if it isn't: why are we caching?)

If you want to go that route, is there a particular reason why you want ICacheSerializer to be generic instead of just its methods being generic? I feel like for the basic scenario of System.Text.Json, this can just be a singleton. If someone needs to differentiate serialization by type it should be easy to do using a composite style pattern.

Instead of configuring caching and serialization by type, I would rather have the ability to have named instances that I can configure in addition to a default instance, similar to how HttpClientFactory works. FusionCache allows this by the way, if you are interested in an example.

[aKzenT]

Instead of configuring caching and serialization by type, I would rather have the ability to have named instances that I can configure in addition to a default instance, similar to how HttpClientFactory works. FusionCache allows this by the way, if you are interested in an example.

Keyed Services might make that easy. Although I'm not sure if there is an easy way to register both a keyed cache and a keyed serializer and tell the DI system that it should use the serializer with the same key when injecting into the cache.

[niemyjski]

I never liked the name of it to start with because I might want an in memory, hybrid or distributed cache and the abstraction doesn't change. I also don't like how expiration / expiration strategy is not present (possibly it is with extension methods, I haven't looked in a long time). I also don't feel very strongly about adding generics to the interface because a cache can store all different kinds of shapes. We probably should slim down our default interface, but this is what we've found useful over the past decade: https://github.com/FoundatioFx/Foundatio#caching

[mgravell]

To add to your list of "Current Code Layout" we have an implementation at AWS for DynamoDB as backend. https://github.com/awslabs/aws-dotnet-distributed-cache-provider

Great to know, thanks! The list was never meant to be exhaustive - I'll add and clarify. The key takeaway is: we want to actively avoid breaking any implementations; the new functionality should happily work against the existing backend, with them using the additional / optional extensions if appropriate.

[jodydonetti]

Preview 4 Notes

Hi @mgravell and team, I played with the preview 4 bits and here are my notes.
They are a mix of 2 things:

• general notes on HybridCache itself, eg: using it as a normal user
• notes about implementing it with FusionCache, eg: using HybridCache as a sort of lingua franca, a shared abstraction that 3rd party packages may depend on (see: OutputCache, etc) while still being able to use different implementations that may have more/different features or scenarios covered

NOTE: I will not touch on things that I know will already change, like RemoveByKey becoming Remove, etc.

At First Glance

I see that there's a SetAsync method but still not a GetAsync method, currently: my first thought (been there, done that etc) is because you may not still be comfortable with exposing the internal "entry" class? If that is the case I would suggest a TryGetAsync method instead, which would still be able to clearly differentiate between null and "no value was there". It requires the definition of a kinda generic "maybe monad" type (in FusionCache it is MaybeValue<T> for example) and considering it's a MS thing maybe would prefer to wait for an "official" general thing. Or maybe I just got it wrong, I don't know 😅

Anyway I'll wait for preview 5 for that.

Regarding the HybridCacheEntryOptions, I see that the way you seem to have made it is that:

• if null is passed, it will use the DefaultEntryOptions as a fallback, if defined, or a somewhat even global-er fallback if even that is not defined. I understand the rationale, but I would suggest for the DefaultEntryOptions on the cache to be defined forcibly instead, as it would remove a lot of issues down the road (just my 2 cents ofc)
• any null property inside of a passed entry options object will fall back to that property's default value on the default entry oprions (or related fallback, see above). Again, I see the rationale, but it can be too limiting: for example in case of props where null is a valid value, and you'll not be able to differentiate between null meaning "fall back to the default" and "I really want null here". Again, just my 2 cents, but my suggestion is to think about this really really carefully, and try to apply Second Order Thinking on this, but like, really hard

As pointed out by some in the video by @Elfocrash , the auto-magic usage of any registered IDistributedCache service in the DI container will:

1. surprise some (because it may have been registered for some other reasons, including the default MemoryDistributedCache one registered by aspnet core automatically)
2. will not be controllable by the user. I would suggest taking the same path as FusionCache (or a similar one anyway) where the user needs to explicitly specify to use one, even if the registered one (in FusionCache there's a method in the builder called WithRegisteredDistributedCache() for example).

Something else that's not clear is what LocalCacheExpiration does or how it works: in the xml docs it's stated that "Cache duration in local cache; when retrieving a cached value from an external cache store, this value will be used to calculate the local cache expiration, not exceeding the remaining overall cache lifetime" and I don't understand if it will be somewhat "overwritten" by some internal logic when getting data from L2, or what. In my tests (again, with preview 4, so things may have already changed...) I saw that with only L1, by setting LocalCacheExpiration I get the expiration I want, whereas by setting Expiration and not setting LocalCacheExpiration I was having surprises. Maybe this is related to one of my points about, meaning in this case it's not possible, by setting LocalCacheExpiration to null, to differentiate between "I don't want to specialize the local cache expiration, just use the normal Expiration" and "for LocalCacheExpiration I want to fall back to a default value". This is subtle but, I'm warning you, very important. Like, very very.

Finally I see that the "by tag" logic is there, all the time: I get that it is desired (and I mean, I really really get it, it would be an awesome feature, and I'm playing with it in FusionCache for some time now already) but, since it is reasonably tied to the distributed backend used, I'm wondering about 2 things:

1. how would existing (and future) implementations of IDistributedCache communicate to HybridCache that they support tagging, or not?
2. what should they do if they do not support that (eg: any one of current SqlServer, CosmosDB, SQLite, etc)? Throw when tags has values? Ignore it? Are there guidelines?
3. how can users of HybridCache check that the current instance actually supports tagging? Is there a public readonly bool prop or something? For example in FusionCache I'm not exposing the underlying IDistributedCache instance, but I'm exposing a bool HasDistributedCache prop to maybe do feature detection, so a similar rational can be applied here
4. how would HybridCache itself knows what to do? Since it's not tied to a specific cache backend (eg: Redis, Memcached, etc) and since there's no concept of a backplane or similar like in FusionCache, what are your thoughts about this? If you can share, I'd like to know

Honestly, I can see the difficulty in creating an API surface area that allows that but also allows to check for that, but still, I'd like to point that out in this first preview.

HybridCache implemented on top of FusionCache

I've been able to reasonably implement very quickly a FusionCache-based adapter adapter for HybridCache, so kudos for how easy it has been.

Right now in the example project I'm able to register the normal HybridCache with this:

// ADD NORMAL HYBRIDCACHE
services.AddHybridCache(o =>
{
	o.DefaultEntryOptions = new HybridCacheEntryOptions
	{
		LocalCacheExpiration = defaultDuration
	};
});

and a FusionCache-based one with this:

// ADD FUSIONCACHE, ALSO USABLE AS HYBRIDCACHE
services.AddFusionCache()
  .AsHybridCache() // MAGIC
  .WithDefaultEntryOptions(o =>
  {
    o.Duration = defaultDuration;
  });

The "magic" part is basically registering a FusionCache-based adapter as an HybridCache service, and the rest of the code is like this:

// THIS WILL WORK BOTH WITH THE DEFAULT HybridCache IMPLEMENTATION
// AND THE FusionCache BASED ONE
var cache = serviceProvider.GetRequiredService<HybridCache>();

Console.WriteLine($"TYPE: {cache.GetType().FullName}");

const string key = "test key";

// TAGS: NOT CURRENTLY SUPPORTED
//await cache.SetAsync("fsdfsdfds", "01", tags: ["", ""]);

// SET
await cache.SetAsync(key, "01");
var foo = await cache.GetOrCreateAsync<string>(
	key,
	async _ => "02"
);
Console.WriteLine($"FOO: {foo} (SHOULD BE 01)");

// REMOVE
await cache.RemoveKeyAsync(key);
foo = await cache.GetOrCreateAsync<string>(
	key,
	async _ => "03"
);
Console.WriteLine($"FOO: {foo} (SHOULD BE 03)");

// NO-OP
foo = await cache.GetOrCreateAsync<string>(
	key,
	async _ => "04"
);
Console.WriteLine($"FOO: {foo} (SHOULD BE 03)");

// LET IT EXPIRE
Console.WriteLine("WAITING FOR EXPIRATION...");
await Task.Delay(defaultDuration);

// SET
foo = await cache.GetOrCreateAsync<string>(
	key,
	async _ => "05"
);
Console.WriteLine($"FOO: {foo} (SHOULD BE 05)");

Running this with both the registrations in turn gives me this first:

TYPE: Microsoft.Extensions.Caching.Hybrid.Internal.DefaultHybridCache
FOO: 01 (SHOULD BE 01)
FOO: 03 (SHOULD BE 03)
FOO: 03 (SHOULD BE 03)
WAITING FOR EXPIRATION...
FOO: 05 (SHOULD BE 05)

and this second:

TYPE: ZiggyCreatures.Caching.Fusion.MicrosoftHybridCache.FusionCacheAdapterForHybridCache
FOO: 01 (SHOULD BE 01)
FOO: 03 (SHOULD BE 03)
FOO: 03 (SHOULD BE 03)
WAITING FOR EXPIRATION...
FOO: 05 (SHOULD BE 05)

Of course right now I'm talking without tagging support, for which I'm currently throwing an exception (because of the reasons I'll highlight soon in the other issue).

Apart from that I think it may be useful to have a method (either protected virtual or public virtual) that allows to "resolve" a single HybridCacheEntryOptions object passed in (including null) to what it will be used in the end, meaning fallbacks and similar.

Currently I'm applying some logic here, simulating what you have described in your specs, but having some cache instance method where you can pass an HybridCacheEntryOptions object (again including null) that will give you back a "normalized" one with all the defaults applied based on whatever logic you'll decide to apply is fundamental, otherwise 3rd party implementers like me will have to manually implement the same logic, again and again.
Of course the problems highlighted above about different meaning for null for props still aply.

Another thing I noticed is that currently it's possible to override the GetOrCreate method that accepts a state, but not the other one without state: from my understanding, this is because the one without state basically calls the other with state but with a logical "null" one.
Again, I understand the rationale, but for libs that do not currently supports explicit state in the methods this means a double closure allocation, because the one with explicit state would call the one without explicit state which would call the one with explicit state. If there is not a hard reason to disallow the override I would suggest allowing it, even though probably not many will do it (but it would still be possible).
PS: I'm working on adding support for explicit state overrides for FusionCache, so I don't know if I will need this, I'm thinking more generally.

I see that the namespace/packages in preview 4 are "wrong" long term: if you are wondering about types forwardings and similar, my suggestion is to just "destroy everything" in the next preview. This is exactly what previews are for, and (again, personally) I'd like to update the packages, see that I'm not able to compile anymore, and know exactly what needs to be changed to allow it to compile again.
It definitely feels better and, in the long run, it pays dividends (again, just my 2 cents).

For now I think it's all, thanks for sharing the update.

Hope this helps.

[mgravell]

Great feedback. I will ponder!

Minor note: there is no additional closure allocation: we pass the stateless callback as the TState, and invoke it from there via a static lambda. This trick is also available to anyone wrapping the library.

[jodydonetti]

Minor note: there is no additional closure allocation: we pass the stateless callback as the TState, and invoke it from there via a static lambda. This trick is also available to anyone wrapping the library.

Oh, that's neat, I didn't notice it!

[HaikAsatryan]

If my application has 2 instances and 1st instance updates record in L1 + L2, will 2nd instance L1 get the updated record as well with some pub/sub magic?

[michael-wolfenden]

Hi @mgravell and team,

LazyCache supports setting the expiry of the cache entry based on the data being cached.

For example:

cache.GetOrAdd("token", async entry => {
    var token = await GetAccessToken();
    // can set expiry date using a DateTimeOffset (or a TimeSpan with RelativeToNow variant)
    entry.AbsoluteExpiration = token.Expires;
    return token;
});

Does HybridCache support this scenario?

[mgravell]

@michael-wolfenden no, the API as implemented doesn't support that option; such options are passed in as arguments, so are fixed at the time of call.

[michael-wolfenden]

Is the team considered adding similar functionality? The api as is stands now doesn't allow the cache time to be set dynamically based on the data being returned which is a common scenario when caching data from a third party where they control the data's lifetime.

[mgravell]

Output-cache already has some inbuilt functionality similar to stampede protection, which I believe (I can check on Monday) is enabled by default. I don't know without checking whether YARP uses output cache. Can you clarify exactly what you are looking for here, so I can check more?

…

On Sat, 29 Jun 2024, 20:48 Tal Wald, ***@***.***> wrote: Hi @mgravell <https://github.com/mgravell>! We are looking to add YARP as a reverse proxy and want the out of the box stampede protection that HybridCache provides. What is the equivalent of UseOutputCache for HybridCache? — Reply to this email directly, view it on GitHub <#53255 (comment)> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAEHMAAC4NON44IVNRH3FTZJ4FSVBFKMF2HI4TJMJ2XIZLTSWBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVIYTEOBUGE3TKMBTGSSG4YLNMWUWQYLTL5WGCYTFNSBKK5TBNR2WLKRVGU3TENBTGEZTAM5ENZQW2ZNJNBQXGX3MMFRGK3FMON2WE2TFMN2F65DZOBS2YSLTON2WKQ3PNVWWK3TUUZ2G64DJMNZZJAVEOR4XAZNKOJSXA33TNF2G64TZUV3GC3DVMWUDCNZWGIYDGNBXQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSGA3TGMRXGE4TMMECUR2HS4DFUVWGCYTFNSSXMYLMOVS2UMJSHA2DCNZVGAZTJAVEOR4XAZNFNRQWEZLMUV3GC3DVMWVDKNJXGI2DGMJTGAZ2O5DSNFTWOZLSUZRXEZLBORSQ> . You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

[vvadyak]

@mgravell and @jodydonetti, I believe combining your efforts will result in a best-in-class cache abstraction

[lindeberg]

Are hit/miss metrics available? How?

[mgravell]

Currently: not; working on it via .net metrics

[stevejgordon]

@mgravell I quickly looked at using HybridCache in a sample app as I was interested in the tiered support and low allocation APIs. Forgive me if it's a very naive question, but the lack of a GetAsync method stumped me for my scenario. I see that it was at one point in the proposal. In the code I'm playing with, I want to access the value from the cache (without creating it). If it's present, I'll handle the case where it's not, but I don't want that to trigger the caching of the result automatically.

In a related scenario, is there a pattern for when the factory can't produce the data (external service is offline), so all we have is some default value we fall back to, such as a static Empty instance or null? For example, if I run this:

var result = await _hybridCache.GetOrCreateAsync("something", _ => ValueTask.FromResult<string?>(null));
result = await _hybridCache.GetOrCreateAsync("something", _ => ValueTask.FromResult<string?>(null));

The factory in the second call won't be invoked as null has been cached.

Ideally, I need a way to signal that the creation failed and not store anything under the key. If there were a way to do this, that might then provide a workaround to the first problem, as I can do GetOrCreate but signal that I don't want anything actually stored in the cache when the entry does not exist.

Another scenario I've yet to explore is where my "expensive" async calls to external systems that I use to build up some data that I'd like to cache, also provide some extra info I need later in my method but that I don't want cached. I'm guessing, but have yet to try, that using the state might be a way to do that but I'm curious if it's a pattern you expect to see used?

[mgravell]

Will do example of the first tomorrow.

On the latter: if you need that value, what do you expect to use on the anticipated case when the value does come from cache? You say you need it, but if it isn't stored: where does it come from?

I guess philosophically it should usually be preferable to have things work the same whether or not the callback was invoked this time, but yes you can always mutate ambient state in either the TState sense or via closures.

Note that there's also a category of cases where the callback was invoked for the active result, but not by you - i.e. a stampede scenario where you're merely an observer.

[stevejgordon]

@mgravell, thanks. This morning, with a fresh eye, I just discovered the flags that seemed to give me the control I needed to avoid the set.

var result = await _hybridCache.GetOrCreateAsync("something", _ => ValueTask.FromResult<string?>(null), 
    new() { Flags = HybridCacheEntryFlags.DisableLocalCacheWrite | HybridCacheEntryFlags.DisableDistributedCacheWrite });

result = await _hybridCache.GetOrCreateAsync("something", _ => ValueTask.FromResult<string?>(null));

The main reason I was looking at this was akin to the FusionCache adaptive caching, where, based on the outcome of the factory, I may want not to cache (or choose a very short cache time) when the external service couldn't give me the value I needed. In that case, I may present an error to the end user (for the current request); for a subsequent request, it tries again to get a value. Using a short empty cached value might be helpful for DDos or general outages to save hitting a struggling backend for a while. However, I may also get that behaviour from Polly and the HTTP resiliency stuff.

If a null/empty value is cached for the configured expiration time, say, 30 mins, that's not ideal. That said, if the cache returns empty, I can still trigger a fetch of the data and manually update the cache. So I think, I see a way to achieve this.

[nkelemen18]

Hello all,

I'm interested in the GetAsync method as well. Can we expect a GetAsync method in the future?

[HugoVG]

I wanted to try the hybrid cache so I've tried implementing it in my current code base, but when I tried to add an EfCore item with cyclic fields(x->y->x->y...) it would crash, It seems to ignore any JsonOptions from the DI, and there was no way to set ReferenceHandler = ReferenceHandler.Preserve; or ReferenceHandler = ReferenceHandler.IgnoreCycles;, are there any plans to change this api or would I have to write a custom serializer for this?

[mgravell]

@HugoVG I'll investigate; fyi this has now moved to dotnet/extensions - I'll log an issue there tomorrow

[HugoVG]

@HugoVG I'll investigate; fyi this has now moved to dotnet/extensions - I'll log an issue there tomorrow

Could you mention me in that one so I could be kept posted. I don't mind help testing

[jamo-semler]

It is a little unclear if we have invalidation of L1 cache across instances (through Redis pub/sub or similar)?

[HaikAsatryan]

It is a little unclear if we have invalidation of L1 cache across instances (through Redis pub/sub or similar)?

Asked same question 3+ months ago noone responded
#53255 (comment)

[mgravell]

Sorry, can't be everywhere at once; short version:

• yes, we absolutely want distributed L1 invalidation
• yes, in the case of Redis that would presumably be via pub/sub, but we want the API to be backend agnostic
• no, we haven't done it yet
• no, it won't be there on day zero

[mdmontesinos]

Is there any expected release date for Hybrid Cache?

[mgravell]

ASAP as soon as it is feature complete; I'm wrapping up the tba-based invalidation right now, which is the main remaining gap.

[mdmontesinos]

@mgravell Great, thanks for the heads up!

[zulander1]

Hello all,

I'm interested in the GetAsync method as well. Can we expect a GetAsync method in the future?

This would be great! Sometimes, all you want to do is check if an object is cached and, if it is, update it. Could we also have an ExistsAsync method?