Non-hermetic DNS testing in eg dynamic_forward_proxy is causing recurring failure/slowness

[phlax]

I stumbled upon this issue with the mobile test //test/common/integration:client_integration_test

in that case the test had a total timeout and was querying dns for doesnotexist - dns was retrying 3x, and taking 5+ seconds to respond on each try - blowing the tests total timeout budget

im struggling to find how i resolved the issue in mobile - but it seems i got past it

this problem has again occurred when trying to update the build image - asan and gcc are reliably timeouting on similar tests - and it looks like for similar reasons

i would have thought these tests should not be calling any real dns - it should be mocked and hermetic

[phlax]

cc @mattklein123 @RyanTheOptimist @jwendell

[phlax]

might be systemd related (see envoyproxy/envoy-build-tools#497) altho that doesnt explain why it would happen when changing the container

also cc @ravenblackx as i see you commenting similar previously

[ravenblackx]

#39613 is a PR where I made a thing use partially mocked DNS functionality. Maybe the mock should be moved out of that specific test file into a helper library, and be used for this case too.

[phlax]

yeah - i think that would be really helpful - im pretty sure this is also the underlying cause we see periodically with dfp flaking

[ravenblackx]

Would be helpful to know which test[s] is/are flaking - turns out #39613 was already fixing a dfp integration test too, it's just that there are several others.

[phlax]

sure i can list the ones ive seen issues with recently

• @envoy_mobile//test/common/integration:client_integration_test
• @envoy//test/extensions/filters/http/dynamic_forward_proxy:proxy_filter_integration_test (asan)
• @envoy//test/extensions/network/dns_resolver/cares:dns_impl_integration_test (gcc)

guessing these are caused by similar issues

[ravenblackx]

Then my thing apparently doesn't resolve it, since proxy_filter_integration_test is the one that's already using the mock dns (unless someone has since added a test path that doesn't).

Edit: looking at the history, #39901 postdates my change and looks like it does do something that alters the DNS behavior, so maybe it's calling some os dns function other than getaddrinfo now.

[phlax]

in the mobile case at least i debugged it and defo was calling getaddrinfo and hanging there - not sure on the envoy ones but they look suspiciously similar