[upgrade] Deb package leaves stale symlink to elastic-agent

[leehinman]

• .deb install
• Debian 11
• upgrade from 8.8.2 -> 8.10.4

After upgrade symlink to elastic-agent is wrong

/var/lib/dpkg/info# ls -l /usr/share/elastic-agent/bin/elastic-agent
lrwxrwxrwx 1 root root 62 Nov 2 12:46 /usr/share/elastic-agent/bin/elastic-agent -> /var/lib/elastic-agent/data/elastic-agent-cdc5ba/elastic-agent

symlink should point to:

/var/lib/elastic-agent/data/elastic-agent-a92ca3/elastic-agent 

looks like elastic-agent.postinst is to blame.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[tibyke]

Just tried it and it is indeed an issue, started of with 8.10.0, installed fine
But then tried upgrading version by version and all failed with the postinstall scriptlet, relevant part:

$ nl -ba /var/lib/dpkg/info/elastic-agent.postinst | sed -n 10,16p
    10  if test -L "$symlink"; then
    11      resolved_symlink="$(readlink -f -- "$symlink")"
    12      # check if it is resolved to non empty string
    13      if ! [ -z "$resolved_symlink" ]; then
    14          old_agent_dir="$( dirname "$resolved_symlink" )"
    15      fi
    16  fi

the problem is that the upgrade already cleans up the destination of the symlink:

$ ll /usr/share/elastic-agent/bin/elastic-agent
lrwxrwxrwx 1 root root 62 Nov 21 16:08 /usr/share/elastic-agent/bin/elastic-agent -> /var/lib/elastic-agent/data/elastic-agent-ab6e68/elastic-agent
$ ll /var/lib/elastic-agent/data/elastic-agent-ab6e68/elastic-agent
ls: cannot access '/var/lib/elastic-agent/data/elastic-agent-ab6e68/elastic-agent': No such file or directory

so the readlink -f will fail (running here with an -vf):

$ readlink -fv /usr/share/elastic-agent/bin/elastic-agent && echo YES || echo NO
readlink: /usr/share/elastic-agent/bin/elastic-agent: No such file or directory
NO

[cmacknz]

Just to clarify, what upgrade command is used to trigger this? dpk --install elastic-agent-$new-amd64.deb or equivalent or elastic-agent upgrade $new? I am assuming the former but it isn't mentioned explicitly.

[tibyke]

the initial command was a sudo apt purge elastic-agent, then an rm -rf to remove she stale symlink.
then installed the first version with sudo apt install elastic-agent=x.yz, that was stil fine.
after that every single sudo apt upgrade elastic-agent=x.y.Z was failing with the aforementioned postinst issue + the left behind stale symlink

[tibyke]

here is the full purge:

$ sudo apt purge -y elastic-agent
[sudo] password for tibyke:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  ieee-data libgnutls-openssl27 libllvm13 libvulkan1 mesa-vulkan-drivers python3-argcomplete python3-libcloud python3-lockfile python3-netaddr python3-pycryptodome python3-selinux
  python3-simplejson
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  elastic-agent*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 1837 MB disk space will be freed.
(Reading database ... 144197 files and directories currently installed.)
Removing elastic-agent (8.11.1) ...
(Reading database ... 140499 files and directories currently installed.)
Purging configuration files for elastic-agent (8.11.1) ...
dpkg: warning: while removing elastic-agent, directory '/usr/share/elastic-agent/bin' not empty so not removed
Scanning processes...

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

after that:

$ tree /usr/share/elastic-agent/
/usr/share/elastic-agent/
└── bin
    └── elastic-agent -> /var/lib/elastic-agent/data/elastic-agent-3c8be7/elastic-agent

1 directory, 1 file

and the target is not there:

$ ll /usr/share/elastic-agent/bin/e*
lrwxrwxrwx 1 root root 62 Nov 21 16:16 /usr/share/elastic-agent/bin/elastic-agent -> /var/lib/elastic-agent/data/elastic-agent-3c8be7/elastic-agent

so manually remove the leftbehind:

$ sudo rm -rfv /usr/share/elastic-agent
removed '/usr/share/elastic-agent/bin/elastic-agent'
removed directory '/usr/share/elastic-agent/bin'
removed directory '/usr/share/elastic-agent'

purge again just to be sure:

$ sudo apt purge -y elastic-agent
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package 'elastic-agent' is not installed, so not removed
The following packages were automatically installed and are no longer required:
  ieee-data libgnutls-openssl27 libllvm13 libvulkan1 mesa-vulkan-drivers python3-argcomplete python3-libcloud python3-lockfile python3-netaddr python3-pycryptodome python3-selinux
  python3-simplejson
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

then install eg. 8.10.0:

$ sudo apt install -y elastic-agent=8.10.0
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  ieee-data libgnutls-openssl27 libllvm13 libvulkan1 mesa-vulkan-drivers python3-argcomplete python3-libcloud python3-lockfile python3-netaddr python3-pycryptodome python3-selinux
  python3-simplejson
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
  elastic-agent
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 552 MB of archives.
After this operation, 1797 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 elastic-agent amd64 8.10.0 [552 MB]
Fetched 552 MB in 1min 18s (7040 kB/s)
Selecting previously unselected package elastic-agent.
(Reading database ... 140493 files and directories currently installed.)
Preparing to unpack .../elastic-agent_8.10.0_amd64.deb ...
Unpacking elastic-agent (8.10.0) ...
Setting up elastic-agent (8.10.0) ...
create symlink /usr/share/elastic-agent/bin/elastic-agent to /var/lib/elastic-agent/data/elastic-agent-471966/elastic-agent
Scanning processes...

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

and then the upgrade fails again:

$ sudo apt upgrade -y elastic-agent=8.10.1
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  ieee-data libgnutls-openssl27 libllvm13 libvulkan1 mesa-vulkan-drivers python3-argcomplete python3-libcloud python3-lockfile python3-netaddr python3-pycryptodome python3-selinux
  python3-simplejson
Use 'sudo apt autoremove' to remove them.
Get more security updates through Ubuntu Pro with 'esm-apps' enabled:
  imagemagick libopenexr25 libmagick++-6.q16-8 libmagickcore-6.q16-6-extra
  libavcodec58 libmagickwand-6.q16-6 libavutil56 imagemagick-6.q16 libswscale5
  libmagickcore-6.q16-6 libswresample3 imagemagick-6-common
Learn more about Ubuntu Pro at https://ubuntu.com/pro
The following packages will be upgraded:
  elastic-agent
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/553 MB of archives.
After this operation, 3379 kB of additional disk space will be used.
(Reading database ... 144198 files and directories currently installed.)
Preparing to unpack .../elastic-agent_8.10.1_amd64.deb ...
Unpacking elastic-agent (8.10.1) over (8.10.0) ...
Setting up elastic-agent (8.10.1) ...
Installing new version of config file /etc/elastic-agent/.elastic-agent.active.commit ...
dpkg: error processing package elastic-agent (--configure):
 installed elastic-agent package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 elastic-agent
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

this is ubuntu 22.04.3

[leehinman]

It looks like just 8.10.X has this issue. upgrading to 8.11.1 worked fine in my testing.

[tibyke]

I tried 2 new scenarios

• initial install of 8.10.4, then upgrade to 8.11.0
• initial install of 8.11.0, then upgrade to 8.11.1

neither of these work as expected, both produce the same symptons as mentioned above:

Installing new version of config file /etc/elastic-agent/.elastic-agent.active.commit ...
dpkg: error processing package elastic-agent (--configure):
 installed elastic-agent package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 elastic-agent
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)