Use polling during startup to wait for informers to sync #6614
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tsaarni
requested review from
skriss and
sunjayBhatia
and removed request for
a team
sunjayBhatia
requested review from
a team,
rajatvig and
izturn
and removed request for
a team
tsaarni
force-pushed
the
poll-hassynced
branch
from
ed7b7de
to
a8ea16b
Compare
tsaarni
added
the
release-note/small
tsaarni
force-pushed
the
poll-hassynced
branch
2 times, most recently
from
3046ee2
to
7693441
Compare
tsaarni
force-pushed
the
poll-hassynced
branch
from
7693441
to
cfcf22d
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #6614 +/- ##
==========================================
+ Coverage 81.01% 81.04% +0.02%
==========================================
Files 133 133
Lines 19997 20001 +4
==========================================
+ Hits 16201 16210 +9
+ Misses 3503 3498 -5
Partials 293 293
|
tsaarni
removed
the
do not merge
|
Fix after test case failure. Coverage target is bit difficult to reach for this one. |
|
The Contour project currently lacks enough contributors to adequately respond to all PRs. This bot triages PRs according to the following rules:
You can:
Please send feedback to the #contour channel in the Kubernetes Slack |
github-actions
bot
added
the
lifecycle/stale
tsaarni
removed
the
lifecycle/stale
|
The Contour project currently lacks enough contributors to adequately respond to all PRs. This bot triages PRs according to the following rules:
You can:
Please send feedback to the #contour channel in the Kubernetes Slack |
github-actions
bot
added
the
lifecycle/stale
tsaarni
removed
the
lifecycle/stale
|
The Contour project currently lacks enough contributors to adequately respond to all PRs. This bot triages PRs according to the following rules:
You can:
Please send feedback to the #contour channel in the Kubernetes Slack |
github-actions
bot
added
the
lifecycle/stale
tsaarni
removed
the
lifecycle/stale
tsaarni
force-pushed
the
poll-hassynced
branch
from
cfcf22d
to
5c542fb
Compare
The "upstream" informer trackers do not always return HasSynced() == true immediately, even if we have already processed all events the informer dispatched us. Change to polling so that we can progress also in that case. Signed-off-by: Tero Saarni <tero.saarni@est.tech>
tsaarni
force-pushed
the
poll-hassynced
branch
from
5c542fb
to
63fcd50
Compare
sunjayBhatia
approved these changes
Oct 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change will use polling for
HasSynced()call to align with how client-go'sWaitForCacheSync()operates. Previously it was only called when we received objects from the informer. By changing to polling, we prevent a race condition where Contour fails to recognize that the final object in the initial list has been received, which would prevent the xDS server from ever starting.Fixes #6613
Details
The startup of a follower instance depends on informers signaling that they have finished synchronizing resources with the API server. We use the
SingleFileTrackerfrom client-go to track the processing of initial objects. This tracker increments when processing starts on a resource and decrements when its finished.The
HasSynced()method should ideally report synchronization statustrueright afterFinished()is called for the last resource in the initial list (tracker reaches zero). However, in some cases it fails to reporttrueimmediately, causig the boolean trigger that starts the xDS server to never be set.In case of a leader instance, an extra update triggered by the leader selection itself has ensured that the xDS server starts.