Elastic-Agent fails to uninstall on Windows and is marked as unupgradable

[belimawr]

For confirmed bugs, please report:

• Version: 8.10.x (I don't recall the minor)
• Operating System: Windows

Elastic-Agent was installed and correctly working on a Windows laptop, the laptop was offline for days and the Elastic Stack it was communicating with was deleted from Elastic Cloud.

Then when trying to uninstall the agent to install a new one it refused to be uninstalled, Windows service manager could not stop the service nor could the task manger kill the process.

Here are the logs from trying to install the new agent:

PS C:\tmp\foo\elastic-agent-8.11.3-windows-x86_64> .\elastic-agent.exe install --url=https://foo.fleet.us-west2.gcp.elastic-cloud.com:443 --enrollment-token=foo==
Error: already installed at: C:\Program Files\Elastic\Agent
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.11/fleet-troubleshooting.html
PS C:\tmp\foo\elastic-agent-8.11.3-windows-x86_64> .\elastic-agent.exe uninstall
Error: can only be uninstalled by executing the installed Elastic Agent at: C:\tmp\foo\elastic-agent-8.11.3-windows-x86_64\elastic-agent.exe
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.11/fleet-troubleshooting.html
PS C:\tmp\foo\elastic-agent-8.11.3-windows-x86_64> C:\tmp\foo\elastic-agent-8.11.3-windows-x86_64\elastic-agent.exe uninstall
Error: can only be uninstalled by executing the installed Elastic Agent at: C:\tmp\foo\elastic-agent-8.11.3-windows-x86_64\elastic-agent.exe
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.11/fleet-troubleshooting.html
PS C:\tmp\foo\elastic-agent-8.11.3-windows-x86_64> .\elastic-agent.exe install --url=https://foo.fleet.us-west2.gcp.elastic-cloud.com:443 --enrollment-token=foo== --force
Uninstalling current Elastic Agent...
   Stopping service................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Trying to uninstall the Elastic-Agent from C:\Program Files\Elastic\Agent also did not work and reported the same error, something like:

PS C:\Program Files\Elastic\Agent> .\elastic-agent.exe uninstall
Error: can only be uninstalled by executing the installed Elastic Agent at: C:\Program Files\Elastic\Agent\elastic-agent.exe
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.11/fleet-troubleshooting.html

Rebooting the machine and uninstalling the Elastic-Agent from C:\Program Files\Elastic\Agent worked.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

We have had this happen in our internal InfoSec deployment. I believe the root cause is a missing .installed marker file. This also prevents you from upgrading the agent from Fleet because being installed is one of the prerequisites for that to work.

elastic-agent/internal/pkg/agent/cmd/uninstall.go

Lines 54 to 57 in 677b79c

	if status == install.Installed && !info.RunningInstalled() {
	return fmt.Errorf("can only be uninstalled by executing the installed Elastic Agent at: %s", install.ExecutablePath(paths.Top()))
	}

elastic-agent/internal/pkg/agent/application/info/state.go

Lines 15 to 30 in 677b79c

	// MarkerFileName is the name of the file that's created by
	// `elastic-agent install` in the Agent's topPath folder to
	// indicate that the Agent executing from the binary under
	// the same topPath folder is an installed Agent.
	const MarkerFileName = ".installed"
	// RunningInstalled returns true when executing Agent is the installed Agent.
	func RunningInstalled() bool {
	// Check if install marker created by `elastic-agent install` exists
	markerFilePath := filepath.Join(paths.Top(), MarkerFileName)
	if _, err := os.Stat(markerFilePath); err != nil {
	return false
	}
	return true
	}

[belimawr]

I tried to reproduce it with a v8.12.0 BC, but it uninstalled when I run the command from

PS C:\Program Files\Elastic\Agent> .\elastic-agent.exe version
Binary: 8.12.0 (build: 7399f1647cb4ceadc419c6ec99e07967eac9e638 at 2023-12-21 12:13:20 +0000 UTC)
Daemon: 8.12.0 (build: 7399f1647cb4ceadc419c6ec99e07967eac9e638 at 2023-12-21 12:13:20 +0000 UTC)
PS C:\Program Files\Elastic\Agent> .\elastic-agent.exe status
┌─ fleet
│  └─ status: (FAILED) fail to checkin to fleet-server: all hosts failed: 1 error occurred:
│         * requester 0/1 to host https://fleet-server:8220/ errored: Post "https://fleet-server:8220/api/fleet/agents/f4430400-89c7-4b52-a3fa-9850722985b5/checkin?": dial tcp 127.0.0.1:8220: connectex: No connection could be made because the target machine actively refused it.
│
│
├─ elastic-agent
│  ├─ status: (DEGRADED) 1 or more components/units in a degraded state
│  └─ endpoint-default
│     ├─ status: (HEALTHY) Healthy: communicating with endpoint service
│     ├─ endpoint-default
│     │  └─ status: (DEGRADED) Applied policy {05ae8b21-9db8-450c-8be8-407088e214e0}
│     └─ endpoint-default-05ae8b21-9db8-450c-8be8-407088e214e0
│        └─ status: (DEGRADED) Applied policy {05ae8b21-9db8-450c-8be8-407088e214e0}
└─ upgrade_details
   ├─ target_version: 8.12.0
   ├─ state: UPG_WATCHING
   ├─ action_id: e05645bc-5896-423e-b7b2-f4183399a16d
   └─ metadata
PS C:\Program Files\Elastic\Agent> .\elastic-agent.exe uninstall
Elastic Agent will be uninstalled from your system at C:\Program Files\Elastic\Agent. Do you want to continue? [Y/n]:y
[====] Failed to uninstall agent  [40s] Error: error uninstalling agent: failed to remove installation directory (C:\Program Files\Elastic\Agent): timed out while removing "C:\\Program Files\\Elastic\\Agent". Last error: remove C:\Program Files\Elastic\Agent\elastic-agent-20231222-10.ndjson: The process cannot access the file because it is being used by another process.
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.12/fleet-troubleshooting.html

The uninstall leaves one log file behind, but it works.

[cmacknz]

Yeah that is a separate bug where the agent still has the file open as we try to delete it.

Here's an explanation of the upgrade marker file:

elastic-agent/internal/pkg/agent/cmd/run.go

Lines 611 to 633 in 7e80290

	func ensureInstallMarkerPresent() error {
	// In v8.8.0, we introduced a new installation marker file to indicate that
	// an Agent was running as installed. When an installed Agent that's older
	// than v8.8.0 is upgraded, this installation marker file is not present.
	// So, in such cases, we need to create it manually post-upgrade.
	// Otherwise, the upgrade will be unsuccessful (see
	// https://github.com/elastic/elastic-agent/issues/2645).
	// Only an installed Elastic Agent can be self-upgraded. So, if the
	// installation marker file is already present, we're all set.
	if info.RunningInstalled() {
	return nil
	}
	// Otherwise, we're being upgraded from a version of an installed Agent
	// that didn't use an installation marker file (that is, before v8.8.0).
	// So create the file now.
	if err := info.CreateInstallMarker(paths.Top(), utils.CurrentFileOwner()); err != nil {
	return fmt.Errorf("unable to create installation marker file during upgrade: %w", err)
	}
	return nil
	}

My main question at this point is what job the .installed marker is doing and whether it is important enough that we should prevent uninstall and upgrade actions from completing if we aren't installed.

[pierrehilbert]

Closing this one in favor of #4051