【BUG】the metric starrocks_be_process_mem_bytes is not precise， maybe casue query Memory of process exceed limit

[kwenzh]

Steps to reproduce the behavior (Required)

1. Construct a large dataset, approximately 300GB,
2. build concurrent queries.
3. watching system resource info
4. check metrics starrocks_be_process_mem_bytes
5. get pod memory usage from /sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd547e58_60f9_4459_a8bb_f0f46043028b.slice/docker-876005925a5ba96dc7e235e0497b7ef832e2befa7f6d60da24aeca141b489df8.scope/memory.usage_in_bytes

Expected behavior (Required)

starrocks_be_process_mem_bytes is near pod mem usage,

Real behavior (Required)

starrocks_be_process_mem_bytes is 48G, but pod memory usage is 35GB,

StarRocks version (Required)

• 4.0.1

mark

and we seem sql query ERROR: Error 1064 (HY000): Memory of process exceed limit. Pipeline Backend: starrocks-be-7.starrocks-be-search.olap.svc.cluster.local, fragment: 019b53bc-1e8b-7095-b66f-53c34122286f Used: 69578920144, Limit: 69578470195. Mem usage has exceed the limit of BE: BE:10343"

so ， I investigated the error return here and found that the memory check comparison and metrics came from the same source. Given the inconsistency between the metrics and actual memory usage, I have questions about whether the calculations were inaccurate, leading to incorrect monitoring metrics and incorrectly aborted queries.

  std::atomic<int64_t> current_value_;
        static const int64_t MAX_INT64 = 9223372036854775807ll;

[kwenzh]

https://github.com/StarRocks/starrocks/blob/4.0.1/be/src/exec/hash_join_components.cpp#L798

Does this fail to consider releasing the parent level's count?

maybe we should be use

_mem_tracker.release(_mem_tracker.consumption()); 

[kwenzh]

This might not be a problem in itself; it's that I used a generated column as the sort key, causing the metrics to spike dramatically. Now that I've switched to a normal column, everything is fine.

`key1` BIGINT NOT NULL DEFAULT "0",
`key2` bigint AS (FLOOR(key1 / 600) * 600),
order by
    (
        `key2`,
        `srcip`
    )

jemalloc

JEMALLOC_CONF: 'percpu_arena:percpu,oversize_threshold:0,muzzy_decay_ms:5000,dirty_decay_ms:5000,metadata_thp:auto,background_thread:true,prof:true,prof_active:false'

However, I still have a question: why does using a generated column exacerbate the increase in starrocks_be_process_mem_bytes and starrocks_be_jemalloc_resident_bytes, when the actual memory usage of the be process isn't increasing?

[stdpain]

@kwenzh If you can find a reproducible case, I believe we can better track this issue.

[kwenzh]

@kwenzh If you can find a reproducible case, I believe we can better track this issue.

Please try writing data to a table where the generated column is used as the sort key.

[stdpain]

not reproduced

[kwenzh]

not reproduced

I understand the crux of the problem now; it's an issue with the Avro library.
If an exception returns that the char * value hasn't been freed, It returned directly within the macro definition.
but this library is no longer maintained, what's your opinion?

https://github.com/apache/avro/blob/release-1.12.0/lang/c/src/value-read.c#L318

[kevincai]

we can patch the library if it is truly the root cause.

[kwenzh]

we can patch the library if it is truly the root cause.

I've confirmed that this library has a problem. In scenarios with abnormal data, such as mismatched schemas and data, it causes a misalignment in the str_len value.

i also find some error log:

Cannot read string value: Cannot read 1699110643 bytes from memory buffer

I believe this library has two flaws:

1. Macros are returned directly without being freed.
2. Length checks are placed after malloc.