bug(batch): restarted serving node occasionally doesn't bump up min_pinned_hummock_version

[zwang28]

Describe the bug

1. A serving node coexists with a frontend node in the same RisingWave standalone process in a pod.
2. Restart the pod.
3. The min_pinned_hummock_version held by this serving node is no longer increasing, unexpectedly. This is the issue.
   • show processlist returns no records.
4. The min_pinned_hummock_version starts to increase due to forceful expiration after max_version_pinning_duration_sec.

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

v1.10

Additional context

No response

[kwannoel]

Restart the pod.

So after pod restarts, somehow after its comes back online, it continues to hold the same min_pinned_hummock_version. Which suggests this is persisted somewhere.

Where do we pin the hummock version, is it managed by meta / serving / frontend?

[zwang28]

Restart the pod.

So after pod restarts, somehow after its comes back online, it continues to hold the same min_pinned_hummock_version. Which suggests this is persisted somewhere.

Where do we pin the hummock version, is it managed by meta / serving / frontend?

Each compute node holds a min_pinned_hummock_version, which is persisted in meta node.
The compute node is expected to bump it up periodically.

I suspect the issue arises from the scan executor in the serving node not being properly canceled after a frontend restart.

1. The frontend schedules execution in serving node.
2. The frontend restarts. Howevert the scheduled execution in serving node is not cancelled accordingly.