Tune or disable discovery rate limiting

[negz]

What problem are you facing?

Like many Kubernetes clients xgql performs 'discovery' - a process via which it walks the API server to determine what APIs it supports. Discovery makes a request per API group, and there are about 350 API groups when the "big three" Crossplane providers are installed with all of their CRDs. xgql's discovery client uses a rate limiter:

xgql/internal/clients/clients.go

Lines 85 to 90 in c6c907a

	func RESTMapper(cfg *rest.Config) (meta.RESTMapper, error) {
	dcfg := rest.CopyConfig(cfg)
	dcfg.QPS = 20
	dcfg.Burst = 100
	return apiutil.NewDynamicRESTMapper(dcfg, apiutil.WithLimiter(rate.NewLimiter(rate.Limit(0.05), 1)))

I believe these rate limits are too low (I also can't remember why we have both dcfg values and a rate.NewLimiter). We've been working with upstream Kubernetes to bump most discovery clients to at least 300qps burst, per kubernetes/kubernetes#109141. There's also a compelling school of thought that we should just disable client-side rate limiting altogether and let server-side API Priority and Fairness let clients know when they should back off.

How could Upbound help solve your problem?

I think we should disable rate-limiting on xgql and try out API Priority and Fairness. If I recall correctly this issue is more of a nuisance than a huge blocker on xgql, because it uses a global discovery client (not one per client) and because it only performs discovery once at startup.