Kolide-Launcher Service Dies on Startup

[prikhi]

Hi, updated my system to latest master on friday & today my kolide-launcher service dies on startup w/ no error level messages:

Sep 25 17:20:14 lovelace systemd[1]: Started The Kolide Launcher.
Sep 25 17:20:14 lovelace kolide-launcher[299853]: {"caller":"main.go:33","msg":"Launcher starting up","revision":"c5af653bd6c065345c6a853a21eabebfb43423c6","severity":"info","ts":"2023-09-25T21:20:14.529705726Z","version":"1.1.0-1-gc5af653b"}
Sep 25 17:20:14 lovelace kolide-launcher[299853]: {"caller":"library_lookup.go:118","component":"tuf_library_lookup","err":"could not initialize TUF client, cannot find release","msg":"could not find executable matching current release","severity":"info","ts":"2023-09-25T21:20:14.529880694Z"}
Sep 25 17:20:14 lovelace kolide-launcher[299853]: {"caller":"logutil.go:13","err":"checking out latest launcher: no versions of launcher in library at /var/lib/kolide/updates","msg":"checking for updated version","severity":"info","ts":"2023-09-25T21:20:14.529950869Z"}
Sep 25 17:20:14 lovelace systemd[1]: kolide-launcher.service: Main process exited, code=exited, status=1/FAILURE
Sep 25 17:20:14 lovelace systemd[1]: kolide-launcher.service: Failed with result 'exit-code'.

The last two log lines also appear in my latest working version so I don't think they are related:

Sep 25 17:20:17 lovelace systemd[1]: Started The Kolide Launcher.
Sep 25 17:20:17 lovelace kolide-launcher[300020]: {"caller":"main.go:33","msg":"Launcher starting up","revision":"d64473a7ae411e5caeffff6fa97abae927245ccf","severity":"info","ts":"2023-09-25T21:20:17.776025795Z","version":"1.0.16-36-gd64473a7"}
Sep 25 17:20:17 lovelace kolide-launcher[300020]: {"caller":"library_lookup.go:107","component":"tuf_library_lookup","err":"could not initialize TUF client, cannot find release","msg":"could not find executable matching current release","severity":"info","ts":"2023-09-25T21:20:17.776096329Z"}
Sep 25 17:20:17 lovelace kolide-launcher[300020]: {"caller":"main.go:188","err":"no versions of launcher in library at /var/lib/kolide/updates","msg":"could not check out latest launcher, will fall back to old autoupdate library","severity":"error","ts":"2023-09-25T21:20:17.776108731Z"}
Sep 25 17:20:17 lovelace kolide-launcher[300020]: {"caller":"main.go:166","msg":"nothing newer","severity":"info","ts":"2023-09-25T21:20:17.776143793Z"}

System info:

$ uname -a
Linux lovelace 6.5.4-arch2-1 #1 SMP PREEMPT_DYNAMIC Thu, 21 Sep 2023 11:06:39 +0000 x86_64 GNU/Linux
$ cat /etc/os-release
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo
$ pacman -Qs osquery
local/osquery 5.7.0-1
    SQL powered operating system instrumentation, monitoring, and analytics

Working version:

$ kolide-launcher --version
{"caller":"main.go:33","msg":"Launcher starting up","revision":"d64473a7ae411e5caeffff6fa97abae927245ccf","severity":"info","ts":"2023-09-25T21:35:08.321406826Z","version":"1.0.16-36-gd64473a7"}
{"caller":"main.go:188","err":"could not get autoupdate config: could not read config file because it does not exist at /etc/kolide-k2/launcher.flags: stat /etc/kolide-k2/launcher.flags: no such file or directory","msg":"could not check out latest launcher, will fall back to old autoupdate library","severity":"error","ts":"2023-09-25T21:35:08.321482262Z"}
{"caller":"main.go:166","msg":"nothing newer","severity":"info","ts":"2023-09-25T21:35:08.321539128Z"}
launcher - version 1.0.16-36-gd64473a7
  branch: 	main
  revision: 	d64473a7ae411e5caeffff6fa97abae927245ccf
  build date: 	2023-09-15
  build user: 	 (prikhi)
  go version: 	go1.21.1

Broken version:

$ kolide-launcher --version
{"caller":"main.go:33","msg":"Launcher starting up","revision":"c5af653bd6c065345c6a853a21eabebfb43423c6","severity":"info","ts":"2023-09-25T21:37:09.515165537Z","version":"1.1.0-1-gc5af653b"}
launcher - version 1.1.0-1-gc5af653b
  branch: 	main
  revision: 	c5af653bd6c065345c6a853a21eabebfb43423c6
  build date: 	2023-09-25
  build user: 	 (prikhi)
  go version: 	go1.21.1

[RebeccaMahany]

@prikhi could you let me know what your update_channel is set to in your launcher.flags file?

[prikhi]

Ah, possibly a bug with the AUR packaging? There is no update_channel config in the provided file: https://aur.archlinux.org/cgit/aur.git/tree/launcher.conf?h=kolide-launcher-git

I build from git instead of relying on auto-updates so I kept the autoupdate line in there commented out

[RebeccaMahany]

I think I know what the issue is -- if you have a moment, could you try setting update_channel to stable, beta, alpha, or nightly in that file and see if the behavior changes? (You can leave autoupdate commented out.)

[prikhi]

It works! Thank you! (I set it to stable because that's what kolide-launcher --help seems to imply the default is.

[RebeccaMahany]

Thanks for confirming! I'll get a fix merged either this evening or tomorrow so you don't need that workaround.