KVStore: Record table-wise memory usage and warn when it goes beyond the threshold

[CalvinNeo]

What problem does this PR solve?

Issue Number: ref #9722

Problem Summary:

• Introduce a struct RegionTableSize which is shared by a Table and all its related Regions. Once registered, a region will update its memory change to this struct, so the table would learn.
• Introduce maybeWarnMemoryLimitByTable and resetWarnMemoryLimitByTable, to log when the memory of a table goes beyond threshold.

Both of the systems works under the path that:

• When a region is created, either for prehandling or inserting, it would register it to RegionTable, and get its shared RegionTableSize.
• Ever since, the insert/delete from the Region would also reflect on this shared RegionTableSize.
• After an insertion, if the memory consumed by a table associated with the given region is reported to go beyond the threshold propotion, and the total memory used is also beyond the given memory limit(which is a very corner case), then we will issue a log.
• Afterwards, if the memory usage somehow goes upper, we don't print the log again for the same table.
• If the memory goes down by committing txns, or applying snapshots, we will reset the warning. So if the table still consumes much memory, a new log will be printed.
• If the region is splitted or merged, the table's memory usage is not affected. If the region is destroyed, the table's memory would be decresed.

Also:

• Remove some useless structures in RegionTable.
• Make safe ts related codes into SafeTsMgr.
• Rename some methods in Region to reflect what they actually do.

What is changed and how it works?

Record table-wise memory usage and warn when it goes beyond threshold

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from calvinneo, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[JaySon-Huang]

Make line 108 in RegionTable::getOrInsertRegion as return insertRegion(table, *region.getRange(), region); and remove this redundant function?

[CalvinNeo]

OK

[JaySon-Huang]

Why we need two method for the same result? Maybe just adding explicit cast when want to assign the result to differnet type?

[CalvinNeo]

total() needs no casting, but is not used very much,
totalSize() needs casting, but is widely used in test, so it is we just write it here to eliminate some static_casting

[JinheLin]

Maybe rename to RegionTableCtxPtr? Most alias of shared_ptr are xxxPtr in TiFlash.

[CalvinNeo]

OK

[CalvinNeo]

Just keep those function name to make them shorter?

[JinheLin]

Just keep those function name to make them shorter?

I think its OK.

[JaySon-Huang]

• If KVStoreMemoryLimit (max_memory_usage_for_all_queries) is explicitly set to 0, now it will not print any logging. IMO, for this case we should use the server memory capacity * 80% as the warning limit.
• Only print log when the region_table_size > 50% * current is too large. I think we may easily miss the logging for track. I recommend to lower the table level memory threshold to min(20% * current, 10GiB) to increase the recall rate for logging down info.

[CalvinNeo]

• OK
• We'd better substract "start_time" memory which could be significant if we set the memory limit to kind of a small value like 4GiB.

[CalvinNeo]

/hold I want to re-perform a test over the new pr.

[CalvinNeo]

Test

[2025/02/14 16:01:37.575 +08:00] [INFO] [Region.cpp:438] ["Memory limit exceeded, current=2756718592 limit=4294967297 table_id=164 keyspace_id=4294967295 region_id=3406 from=put"] [thread_id=181]

When rss is about 3.63 GiB, the KVStore memory consumption is about 619MB, which is about 16.7% of the rss.

The reported point is at 2629MB.

[CalvinNeo]

Another test that prints the table size

[2025/02/14 16:37:01.643 +08:00] [INFO] [Region.cpp:439] ["Memory limit exceeded, current=2866593792 limit=4294967297 table_size=143347752 table_id=168 keyspace_id=4294967295 region_id=3458 from=put"] [thread_id=353]

So the table size is 143347752 bytes(137MiB), while the current is 2866593792 bytes(2.66G), about 5%.

It shows the KVStore size is 630MB, while the usage is 3.74GiB, about 16%.

Consider the base memory usage is 2.37 GiB. It shows a byte cost in KVStore will result in 2 bytes cost in memory usage.

• 137MiB(by log) vs 0.29GiB 46.9%
• 630MB(by grafana) vs 1.37GiB 44.9%

Suppose the memory starts from X, and grows to Y, which is our "warning limit". And if it is all due to one table in KVStore, the table's "table_size" should be (Y - X) / 2.

[CalvinNeo]

New method

[2025/02/14 18:59:53.199 +08:00] [INFO] [Server.cpp:975] ["Global memory status: computed_limit 4294967297 tranquil_time_rss 737959936, cur_virt_size 2888970240, capacity 403755778048"] [thread_id=1]

[2025/02/14 19:09:16.426 +08:00] [INFO] [Region.cpp:444] ["Memory limit exceeded, current=3889836032 limit=4294967297 limit_size=945562828.8 table_size=641429928 table_id=183 keyspace_id=4294967295 region_id=4226 from=put"] [thread_id=368]