Entity storage backend

[moltar]

Today I realized that Redis was used for entity storage and was quite shocked. Redis cannot be a reliable and durable store for entities. At best, it can be a caching layer.

I understand that Redis can save to disk, but it's still not as reliable as an RDBMS.

Was there a particular technical decision that led to this?

Are there any plans to provide a more durable storage layer, such as PostgreSQL?

Thanks.

[alexluong]

Hi there!

It was an intentional technical decision from our side to use Redis for entity storage. With the right configuration, we believe it can be highly available and reliable, similar to RDBMS.

We decided to decouple entity storage & log storage, using different technology, learning from our experience building Hookdeck, specifically around the log storage. We intended to use ClickHouse for that purpose and decided that Redis can be a reliable option for entity storage as well as other KV usage to simplify the stack. The alternative would have been Redis for KV, Postgres for entity, and ClickHouse for log storage, which is doable but a bit more complex than necessary.

With that in mind, supporting another option such as PostgreSQL for entity storage is something we considered but it's currently not on our immediate roadmap. If this is something you and the community strongly prefer, we can consider and re-prioritize accordingly.

Hope that gives you some insights into our decision making and please let us know if you have anything else in mind. We'd love to learn more if you have resources documenting Redis shortcomings or issues with its reliability.

[moltar]

Thanks, @alexluong!

[alexbouchardd]

Just to clarify our thinking here, Outpost was really designed to operate without an RDBMS at all. The ease of adoption of PG led us to implement support for it first, but Clickhouse is in the works and will be considered the "recommended" storage adapter for any serious high-throughput use cases. We've learn at Hookdeck that PG doesn't scale efficiently for such write-heavy workloads. Avoiding an RDMS simplifies horizontal scaling capacity drastically (no connection pooling issues), reduces p99 latency (no need to hydrate the cache data), reduces complexity (cache invalidation) and is more efficient (lower cost). Ultimately, it also just doesn't need it.

Redis with AOF is strongly durable, and RDB is "close enough" with a short snapshot window. Redis is used to store destination configurations which change rarely. Practically speaking, with 1-2 multi-az replicas and RDB on ~1m window, the risk of significant data loss is functionally equivalent to PG.