Getting core dump in ECS using Bottlerocket OS

[josh-kim-resolver]

I'm trying to configure Bottlerocket OS to generate core dumps when containers crash in ECS. My goal is to create a bind mount between the host and the container so that the core dumps are preserved after a crash. To do this, I need the core dumps to be saved in a specific directory where the bind mount is created. However, I'm facing an issue configuring /proc/sys/kernel/core_pattern on the host, which is used to specify the core dump location, because it's read-only and I can't find a way to work around this limitation.

I read about bootstrap containers in the documentation and am wondering if they could be used to modify the core_pattern file. Any help or guidance on how to get core dumps in ECS using Bottlerocket OS would be greatly appreciated. Thanks!

[cloudwitch]

Also need core dumps for EKS and ECS.

I'm using a similar pattern to @josh-kim-resolver where the OS packages the dump into a tarball and my core dump handler container ships that to S3 via watching inotify.

[KCSesh]

Hey @josh-kim-resolver and @cloudwitch,

You should be able to update this value as apart of the Bottlerocket Settings, specifically here:
https://bottlerocket.dev/en/os/1.30.x/api/settings/kernel/#sysctl

You can either run the command directly:

apiclient set --json '
{
  "kernel": {
    "sysctl": {
      "kernel.core_pattern": "/tmp/core.%e.%p.%h.%t"
    }
  }
}
'

Which would then require a reboot.

Or you could set this value in the user-data of your instance, ensuring the value is set from the beginning.

[settings.kernel.sysctl]
"kernel.core_pattern" = "/tmp/core.%e.%p.%h.%t"

[cloudwitch]

I can cause dumps of sleep from within the admin container and see them being created. But I seem not to be able to create a dump from within a container running in a Kubernetes Pod (root or non-root) or outside the admin container via the SSM User. Do you have any ideas as to why this would be?

[cloudwitch]

root@core-dump-handler-fiona-testing-rlzq5:/core_dump_handler# sleep 100 &
[1] 132
root@core-dump-handler-fiona-testing-rlzq5:/core_dump_handler# kill -QUIT 132
root@core-dump-handler-fiona-testing-rlzq5:/core_dump_handler#
[1]+  Quit                    sleep 100
root@core-dump-handler-fiona-testing-rlzq5:/core_dump_handler# cat /proc/sys/kernel/core_pattern
/.bottlerocket/rootfs/var/lib/core_dumps/core-%e-%t-%p-%s
root@core-dump-handler-fiona-testing-rlzq5:/core_dump_handler# echo /core_dumps/core-%e-%t-%p-%s > /proc/sys/kernel/core_pattern
root@core-dump-handler-fiona-testing-rlzq5:/core_dump_handler# cat /proc/sys/kernel/core_pattern
/core_dumps/core-%e-%t-%p-%s
root@core-dump-handler-fiona-testing-rlzq5:/core_dump_handler# sleep 100 &
[1] 1430
root@core-dump-handler-fiona-testing-rlzq5:/core_dump_handler# kill -QUIT 1430
root@core-dump-handler-fiona-testing-rlzq5:/core_dump_handler#
[1]+  Quit                    (core dumped) sleep 100
root@core-dump-handler-fiona-testing-rlzq5:/core_dump_handler#

I have to have a different path inside the host and inside the container. 🤦‍♀️

[cloudwitch]

So this begs the question, how should I set this up correctly?

When I try to set the sysctl in securityContext for the k8s daemonset, I get SysctlForbidden.

      securityContext:
        sysctls:
          - name: kernel.core_pattern
            value: "/core_dumps/core-%e-%t-%p-%

[cloudwitch]

The dumps inside the containers that are doing the dump have to exist in the container. So that means any container I want to dump must also have the path created inside the dump.

So if I don't have /var/lib/core_dumps as a path inside the container, it won't dump.

But if I have /core_dumps in the container, then that mapped to /var/lib/core_dumps in the system, then the pattern /core_dumps/core-%e-%t-%p-%s works for the container, and ends up landing at /var/lib/core_dumps on the EBS volume.

Annoying, but I know my path forward. I will simply mount /var/lib/core_dumps appropriately inside the container as a volume.

Now I need to test something crashing for real to see where that dump actually lands.

[cloudwitch]

I had to modify my PSS to privileged on the namespace the pod was running in. Then mount the /var/lib/core_dumps to /core_dumps/, pattern set to /core_dumps/core-%e-%t-%p-%s in the userdata TOML.

And everything works.

Not ideal because I didn't have to do any of that when running on Amazon Linux 2. But this does indeed work.

I'd prefer the way it worked on AL2, but I'll take what I can get at this point. I assume this really comes down to Bottlerocket OS being a stripped down OS which will introduce quirks for better or worse.

[cloudwitch]

Currently working on this and hitting an issue... I can't take a dump.

I'm using a bootstrap container to create a directory. I was going for /core_dumps, but had to settle on /var/lib/core_dumps because I was struggling with getting the directory on the root filesystem. Below is the script I use in an Alpine container. It works, /.bottlerocket/rootfs/var/lib/core_dumps is created on the host with 777 permissions (I need it wide open so my core dump handler pod can handle the core dumps).

I use kubectl exec to get into the non-root dump handler pod, start up top, in another terminal kubectl exec into the same pod, run kill -11 <pid of top>. This would have created a dump on Amazon Linux 2. I also tried kill -6 without luck. I tried from the admin container too and same results.

Is there a trick to creating a dump?

Edit: kill -QUIT <pid> worked with sleep from the admin container with the pattern set as /tmp/core-%e-%t-%p-%s. Still testing from other containers and my desired path.

Edit2: Needs the full path. /.bottlerocket/rootfs/var/lib/core_dumps/core-%e-%t-%p-%s worked when /var/lib/core_dumps/core-%e-%t-%p-%s would not, likely due to that not being a real path.

#!/bin/sh
set -e

echo "Creating /core_dumps on Host"
mkdir -p /.bottlerocket/rootfs/var/lib/core_dumps || {
  echo "Failed to create directory"
  exit 1
}

chmod 777 /.bottlerocket/rootfs/var/lib/core_dumps || {
  echo "Failed to set permissions"
  exit 1
}

echo "Successfully created and configured /core_dumps directory"

Here is the userdata TOML I'm using in case anyone is struggling.

Edit2: Updated the pattern to be correct.

[settings.bootstrap-containers.core-dump-init]
source = "someaccountnumber.dkr.ecr.some-region-3.amazonaws.com/my/fancy/path/bottlerocket-core-dump-init:sometag"
mode = "always"
[settings.kernel.sysctl]
"kernel.core_pattern" = "/.bottlerocket/rootfs/var/lib/core_dumps/core-%e-%t-%p-%s"

[cloudwitch]

/var/lib/core_dumps/core-%e-%t-%p-%s isn't working for some reason. Switching to /.bottlerocket/rootfs/var/lib/core_dumps/core-%e-%t-%p-%s. Maybe it needs the full path?

[cloudwitch]

Yes! That's the ticket!!! It needs the full path!!!