Forward HTTP_PROXY/HTTPS_PROXY/NO_PROXY to components.

[AndersonQ]

Describe the enhancement:

Currently there are 3 different proxies that can be configured for the Elastic Agent:

• proxy to communicate with ES
• proxy to communicate with Fleet Server
• a default proxy.

The default proxy is set though the environment variables HTTP_PROXY/HTTPS_PROXY/NO_PROXY. This is actually a Go feature, we don't handle them explicitly, but any other proxy will override it.

The proxy setting doe thorough HTTP_PROXY/HTTPS_PROXY/NO_PROXY are inherited by the components the Elastic Agent runs as sub process, that means the beats "receive" those settings when they're set for the Elastic Agent, but other components don't. What is at least an inconsistent behaviour.

Describe a specific use case for the enhancement or feature:

A user using the environment variables to set up a global proxy and having problems because some of the Elastic Agent components, such as Endpoint Security, aren't receiving the proxy.

[mbudge]

Wouldn’t it be better if this was ELASTIC_HTTPS_PROXY so we can manage the proxy when elastic agent is deployed, without other proxies overriding the setting.

The proxy env variables could be created by elastic agent through command line parameters.

elastic/elastic-stack-installers#212 (comment)

[cmacknz]

Setting HTTPS_PROXY or HTTP_PROXY will set the proxy for every connection made by Elastic Agent, except those that originate from Elastic Defend which runs as an independent protected service and so doesn't inherit this environment variable. Correcting this inconsistency where the _PROXY environment variables don't affect Elastic Defend is what this issue is primarily about.

You can also control the proxy for each individual connection (Fleet, Elasticsearch or Output, and Upgrade downloads) independently directly in the agent policy.

Unless I'm missing something here, I don't know why we'd introduce another proxy environment variable.

[mbudge]

Ignore me the proxy stuff with fleet worked fine.

[amitkanfer]

@cmacknz @AndersonQ what's the action item here?

[cmacknz]

It would be forward the value of the *_PROXY environment variables set on the elastic-agent process to the endpoint-security process using the control protocol, so that users don't have to set them in two places.

[pierrehilbert]

@cmacknz do you have an estimate for this one? I would like to plan it in our next coming sprint.

[intxgo]

It would be forward the value of the *_PROXY environment variables set on the elastic-agent process to the endpoint-security process using the control protocol, so that users don't have to set them in two places.

Yes, it was already discussed, but some technical dependencies were holding us back then at Endpoint side. We're already ready to proceed with it. It'll be quite straightforward to implement, the foundation is already there in Elastic Defend sources.

[cmacknz]

We need to confirm and document the precedence order of the proxy configuration fields as part of this. That is if endpoint gets the contents of the HTTPS_PROXY environment variable and also the proxy_url key is set on the output, which one does it prefer?

[amitkanfer]

@cmacknz can / should we compile the effective proxy in fleet / agent and pass only that to endpoint?

[jlind23]

We need to confirm and document the precedence order of the proxy configuration fields as part of this. That is if endpoint gets the contents of the HTTPS_PROXY environment variable and also the proxy_url key is set on the output, which one does it prefer?

I guess they shouldn't look up for environment variables when it comes to proxy settings and only rely on values sent down though the control protocol.
@ferullo this is what we discussed about yesterday right?

[ferullo]

Yeah, it makes sense to me that Endpoint won't consider environment variables at all once this is implemented, but I'll defer to @intxgo and @nfritts on that.

[cmacknz]

I guess they shouldn't look up for environment variables when it comes to proxy settings and only rely on values sent down though the control protocol.

That is the expectation yes. For the variable precedence, we need to test and document this from the agent perspective so that it knows what it should be sending to endpoint.

The HTTPS_PROXY and related variables are features of Go, we do not test them and only document that they exist. There is no documentation explaining how it interacts with the individual proxy_url variables yet.

@cmacknz can / should we compile the effective proxy in fleet / agent and pass only that to endpoint?

We could, I like the simplicity of this as from endpoint's perspective there are no code changes if they already respect the individual proxy_url variables. As mentioned earlier we just need to make sure we use the right precedence rules.

The only thing I don't like about sending the effective proxy is that the output and fleet configuration sent to endpoint no longer matches what is in the agent policy. One way to solve this would be to keep proxy_url as the effective proxy, but also include the other values used to compute it with new keys that are informational only. For example we could put the value of HTTPS_PROXY in an env_https_proxy_url and the one from the policy as policy_proxy_url or something so that looking at the endpoint output unit still has all the information needed to debug proxy behavior.

[intxgo]

regarding the precedence rules, we've already implemented that on Endpoint side. Native proxy setting from policy is taken first, if it's not set, when we fallback from proxy from agent (which we don't have yet).

[cmacknz]

Doing a quick test locally, on the side agent it looks like if I set both HTTPS_PROXY or HTTP_PROXY and the proxy_url for a given connection in the policy at the same time, then the proxy_url in the policy takes precedence.

[intxgo]

Sounds great, IMO as a user I would expect Agent/Endpoint to use what's configured explicitly in the policy rather than preferring some "random" system settings

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[jlind23]

@michel-laterman @ycombinator this issue status has been changed to implementation but there is no PR attached to it, is this really started?

[michel-laterman]

There's no pr yet

[michel-laterman]

@cmacknz, I have a draft for the change and @blakerouse rouse some concerns about the intended scope of the change.
Should we inject the proxy_url value into every output that does not have it, or do we want to limit the scope to endpoint or endpoint+elasticsearch?

[cmacknz]

Only providing this to endpoint is fine as a starting point, they are the only component that will actually use it today.

[cmacknz]

@michel-laterman can you describe a test scenario for a manual test scenario so we can confirm this inter-operates with endpoint as expected?

If it is easier an automated test could be added, but I want to make sure there has been a spot check that endpoint actually gets the proxy and uses it as expected.

[elasticmachine]

Pinging @elastic/fleet-qasource-external (Team:Fleet-QA)

[michel-laterman]

Certainly!

When setting up the test environment, there must be a proxy in between the instance that has the elastic-agent installed and the deployment.

To test:
The elastic-agent should only have the default proxy options specified, see: https://www.elastic.co/guide/en/fleet/current/host-proxy-env-vars.html
Enroll the elastic-agent with a policy with the endpoint integration.

endpoint data from the instance should be available in ES.
proxy_url variables should be injected and appear in endpoint's component configs (in the output and fleet blocks)

[amolnater-qasource]

Thank you for the update Team.
We will test this once 8.15.0 BC2 is available and will let you know in case we had any queries.

[intxgo]

proxy_url variables should be injected and appear in endpoint's component configs (in the output and fleet blocks)

for complete testing please run inspect command on Endpoint to ensure that the proxy_url appears in the relevant sections, to confirm that Agent's view of the policy has been exchanged with Endpoint

Note: be patient, it takes some seconds for new Agent process to re-establish connection with Endpoint to exchange the policy change. This however is not visible in the inspect output as the policy revision stays unchanged matching the revision known by the Elatic stack. You can use status command on elastic-endpoint.exe to observe when the connection with Agent changes to connected status before running inspect.

BTW, status and inspect commands on Endpoint side are added in 8.15. The public documentation for it is in development

cc @amolnater-qasource

[intxgo]

I've tested it locally on Windows, Agent env proxy is applied on Endpoint policy only if the policy doesn't have a proxy defined yet. 👍
Changing proxy propagates to Endpoint at each Agent restart. 👍
Removing it by setting the env variable to empty string removes the proxy from Endpoint policy. 👍

[amolnater-qasource]

Hi Team,

Thank you @intxgo for helping out in testing this.

We have attempted to test this feature on 8.15.0 BC2 and had below observations:

• We were testing on MAC agent+endpoint using the mitmproxy.
• We added the proxy under Fleet>Settings: https://localhost:8080
• Certificate was added with default Elasticsearch output too.
  
  
• Agent was working fine with these settings and no errors were observed.

Endpoint Inspect till here:
Inspect.txt

• Then we updated the /Library/LaunchDaemons/co.elastic.elastic-agent.plist with below configuration:
  
• Then on service restart we are getting error under installed agent:
  

Further as per security reasons, we can't install proxy on AWS VMs since their IPs are exposed.
Hence, we were trying the same on Hardware MAC machines.

Please let us know if we are missing anything from our end.
Thanks!

[intxgo]

Thanks for logging the findings, but that's really not the whole picture from our earlier ad-hoc slack discussion. To recap:

You've applied env proxy on Agent. After restarting Agent it couldn't communicate with Fleet. There was no Fleet proxy in the policy, so it used the env proxy, but the custom self-signed certificate was not provided in the policy for Fleet config, only for Output.

The settings forwarded to Endpoint, as confrimed by elastic-endpoint inspect were:

1. Output config section contained the proxy_url and the custom certificate setting
2. Fleet config did not contain the proxy_url

Actually p.2 is an Agent bug, the design is that env proxy settings should override, both on Agent and Endpoint, the fleet and output sections provided that no proxy is configured explicitly in policy for the respective sections. Agent couldn't communicate with fleet anymore after restart so it appears that it applied the env proxy to fleet config (couldn't communicate because of the lack of custom CA), but it didn't inject the proxy into Endpoint's policy fleet section.

To further clarify, the setting just creates a proxy config entry:

We added the proxy under Fleet>Settings: https://localhost:8080

which has to be explicitly assigned to Output and/or Fleet config. For the sake of testing env proxy, after it has been assigned to Output and Fleet in the policy env proxy should have been set to some bogus unreachable value to test that it's not being used because the proxy settings are explicitly set in the policy (and both Agent and Endpoint sill communicate well with the world using proxy form the policy).

[intxgo]

@aleksmaus we might double-check the fleet proxy override