Block replication and query RPC calls until wait for dba grants has completed

[GuptaManan100]

Description

This PR tries to fix the issue pointed out in #14834.
This is an extension of #14565 and #14680.

The proposed fix in this PR is to block all the replication and query based RPC calls from running until the waitForDBAGrants has succeeded. To this end, we have introduced a new member in the tabletmanager struct which is a channel that we use to know if the function call has finished or not.

Since waitForDBAGrants was a function only being used in the tabletmanager package, we have moved it there and made it a local function. This is also good because it now needs access to the newly added channel variable.

This change ensures that we have verified and waited for the users to have the correct grants when we start running replication and query RPC calls.

This change has the added benefit of not re-introducing the problem in #14681

Related Issue(s)

• Fixes Bug Report: RPCs to a new started VTTablet sometimes fail #14834

Checklist

• "Backport to:" labels have been added if this change should be back-ported to release branches
• If this change is to be back-ported to previous releases, a justification is included in the PR description
• Tests were added or are not required
• Did the new or modified tests pass consistently locally and on CI?
• Documentation was added or is not required

Deployment Notes

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[mattlord]

I'm not sure how I feel about this solution. Would it make sense to handle/wait for this initialization work here?

vitess/go/vt/vttablet/tabletserver/connpool/pool.go

Lines 67 to 96 in 04028a9

	// NewPool creates a new Pool. The name is used
	// to publish stats only.
	func NewPool(env tabletenv.Env, name string, cfg tabletenv.ConnPoolConfig) *Pool {
	cp := &Pool{
	timeout: cfg.Timeout,
	env: env,
	}
	config := smartconnpool.Config[*Conn]{
	Capacity: int64(cfg.Size),
	IdleTimeout: cfg.IdleTimeout,
	MaxLifetime: cfg.MaxLifetime,
	RefreshInterval: mysqlctl.PoolDynamicHostnameResolution,
	}
	if name != "" {
	config.LogWait = func(start time.Time) {
	env.Stats().WaitTimings.Record(name+"ResourceWaitTime", start)
	}
	cp.getConnTime = env.Exporter().NewTimings(name+"GetConnTime", "Tracks the amount of time it takes to get a connection", "Settings")
	}
	cp.ConnPool = smartconnpool.NewPool(&config)
	cp.ConnPool.RegisterStats(env.Exporter(), name)
	cp.dbaPool = dbconnpool.NewConnectionPool("", env.Exporter(), 1, config.IdleTimeout, config.MaxLifetime, 0)
	return cp
	}

[GuptaManan100]

The thing with NewPool is that it just initializes the configurations of the pool and doesn't actually create a connection. So, you can create the pool without needing to wait for anything. We need to wait when the connections are created. That's why I added it to the RPCs. But I agree with you @mattlord, I am just as apprehensive of this solution. If there is a better alternate, I would love to change this implmentation.

[dbussink]

As discussed, I think the alternatives considered are worse than this solution. Guess the least terrible solution for now.