Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added preStop hook to DB statefulset #1263

Merged
merged 1 commit into from
Dec 11, 2023
Merged

Added preStop hook to DB statefulset #1263

merged 1 commit into from
Dec 11, 2023

Conversation

dannyzaken
Copy link
Member

@dannyzaken dannyzaken commented Dec 6, 2023 •
edited
Loading

Explain the changes

  1. The preStop hook will will be executed before terminating the pod. Will call the following command: 
    pg_ctl -D /var/lib/pgsql/data/userdata/ -w -t 60 -m fast stop
  2. -m fast will cause a rollback of all active transactions and stop the server. by default postgres will wait for all active connections to complete before exiting.
  3. the default grace period of Kubernetes for pod shutdown is 30 seconds, and after that, it sends SIGKILL to the process. If there are long-running pending queries, this can lead to WAL corruption

Issues: Fixed #xxx / Gap #xxx

  1. https://bugzilla.redhat.com/show_bug.cgi?id=2165907

Testing Instructions:

  • Doc added/updated
  • Tests added

@tangledbytes
Copy link
Member

One more inconsistency that we might want to consider fixing some time in the future is when we perform several DB ops and expect them to run after one another. Sometimes even when we can succefully restore a DB, one thing that concerns me is partial table writes (for example table1 write completed but update to table2 didn't make it) leaving a logical inconsistency. We might want to start using transactions more frequently than we do right now, WDYT?

@dannyzaken
Copy link
Member Author

@tangledbytes I agree regarding using transactions in more places.
There is this PR by @aspandey which is meant to do exactly that for system store make_changes. It was kind of forgotten and we should probably refresh it.

regardless, I think this is not something that is introduced by this PR, and the logical inconsistencies can also occur with the current behavior

@dannyzaken
added preStop hook to DB statefulset
92e187c 
The preStop hook will call pg_ctl with -m fast flag - this will rollback all active transactions and stop the server.
by default postgres will wait for all active connections to complete before exiting. the default grace period of kuberentes for pod shutdown is 30 seconds and after that it sends SIGKILL to the process. this can lead to DB failures

Signed-off-by: Danny Zaken <dannyzaken@gmail.com>
@dannyzaken dannyzaken merged commit 8b1e465 into noobaa:master Dec 11, 2023
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tangledbytes tangledbytes tangledbytes approved these changes

@liranmauda liranmauda Awaiting requested review from liranmauda

@nimrod-becker nimrod-becker Awaiting requested review from nimrod-becker

Assignees
No one assigned
Labels
size/XS
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dannyzaken @tangledbytes