-
Notifications
You must be signed in to change notification settings - Fork 365
Description
Add a description
I have a case where we need to cache a big table with thousands of rows in it but in cache we need only hundreds of them filtering by some flag (roller = 1). In case we will specify this in kWhere we gonna loose incremental updates for rows which were disabled (roller = 1 -> roller = 0) and this update gonna be received only on full update.
Another approach is to keep kWhere empty and filter rows in CacheInsertOrAssign method. This will solve the problem of incremental updates but for full updates we need to send a "heavy" query to db to select the full table (thousands of rows) and skip almost all of them on the service side.
To solve both of this problems I suggest to add one more field in CachePolicy: kDeltaWhere which gonna override kWhere in case it is specified:
template <typename PostgreCachePolicy>
std::string PostgreCache<PostgreCachePolicy>::GetDeltaWhereClause() {
if constexpr (pg_cache::detail::kHasDeltaWhere<PostgreCachePolicy>) {
// Use kDeltaWhere if explicitly defined
if constexpr (std::string_view{PostgreCachePolicy::kDeltaWhere}.empty()) {
// Empty kDeltaWhere means no additional filtering in delta queries
return fmt::format(FMT_COMPILE("where {} >= $1"), PostgreCachePolicy::kUpdatedField);
} else {
return fmt::format(
FMT_COMPILE("where ({}) and {} >= $1"), PostgreCachePolicy::kDeltaWhere, PostgreCachePolicy::kUpdatedField
);
}
} else if constexpr (pg_cache::detail::kHasWhere<PostgreCachePolicy>) {
// Fall back to kWhere
return fmt::format(
FMT_COMPILE("where ({}) and {} >= $1"), PostgreCachePolicy::kWhere, PostgreCachePolicy::kUpdatedField
);
} else {
return fmt::format(FMT_COMPILE("where {} >= $1"), PostgreCachePolicy::kUpdatedField);
}
}With this logic we could use this CachePolicy:
struct MyCachePolicy {
....
// Filter for full updates: only fetch active rows
static constexpr const char* kWhere = "roller = 1";
// For delta updates, fetch all changed records (including disabled) so CacheInsertOrAssign can handle removals
static constexpr const char* kDeltaWhere = "";
};And this CacheInsertOrAssign function:
template <typename KeyMember>
void CacheInsertOrAssign(
Container& container,
Banner&& value,
const KeyMember& key_member
) {
auto key = std::invoke(key_member, value);
auto it = container.find(key);
if (value.roller == 1) {
// Insert or update
} else {
// Remove from cache if exists
if (it != index.end()) {
container.erase(it);
}
}
}With this logic we will send "light" query for full update, selecting only active rows and for incremental updates the query will also select only few updated rows.
This change won't brake old caches because new logic applies only in case kDeltaWhere is specified in cache policy.