failed to build custom image for RPI4

[fafarun]

Hi,

I tried to build custom image for RPI4, but I hase some issues.
My RPI4 node is used a 1To SSD disk (usb-data connector).

I follow the documentation : https://kairos.io/docs/installation/raspberry/
In the example, the partition size are too small to start k3s server.

I tried to change this value, with a cloud-init config like here :
But nothing append on the node.
Where Can I find logs about cloud-init on kairos ?

Next, I tried to change the value in the osbuilder-tools : like this

I/O size (minimum/optimal): 512 bytes / 512 bytes

>>> Script header accepted.
#!/bin/bash

#download kairosctl
curl -L https://github.com/kairos-io/provider-kairos/releases/download/v2.2.1/kairosctl-.v2.2.1-.linux-.amd64.tar.gz -o - | tar -xvzf - -C .
sudo install kairosctl /usr/bin/kairosctl

# Download the Kairos image locally
#IMAGE=quay.io/kairos/kairos-alpine-arm-rpi-img:v2.3.1-k3sv1.27.3-k3s1
IMAGE=quay.io/kairos/kairos-opensuse-leap-arm-rpi:v2.2.1-k3sv1.26.4-k3s1
docker pull $IMAGE
# Validate the configuration file
kairosctl validate cloud-config.yaml
# Customize it
mkdir -p build
docker run -v $PWD:/HERE \
 -v /var/run/docker.sock:/var/run/docker.sock \
 --privileged -i --rm \
 --entrypoint=/build-arm-image.sh quay.io/kairos/osbuilder-tools:v0.8.6 \
 --use-lvm \
 --model rpi4 \
 --state-partition-size 26200 \
 --recovery-partition-size 4200 \
 --size 15200 \
 --images-size 200000 \
 --local \
 --config /HERE/cloud-config.yaml \
 --docker-image $IMAGE /HERE/build/out.img

But with this configuration, the build job failed :

New situation:
Disklabel type: gpt
Disk identifier: C726A424-964A-8E4A-87CF-E26C34151BEE

The partition table has been altered.
Syncing disks.
+ sgdisk -n 1:8192:+96M -c 1:EFI -t 1:0c00 /HERE/build/out.img
Setting name!
partNum is 0
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
+ sgdisk -n 2:0:+26200M -c 2:state -t 2:8300 /HERE/build/out.img
Setting name!
partNum is 1
Could not create partition 2 from 204800 to 53862399
Unable to set partition 2's name to 'state'!
Could not change partition 2's type code to 8300!
Error encountered; not saving changes.
+ cleanup
+ sync
+ sync
+ sleep 5
+ sync
+ '[' -n /tmp/arm-builder.UDnCTvoMXS ']'
+ rm -rf /tmp/arm-builder.UDnCTvoMXS
+ '[' -n /tmp/arm-builder.wINEuGvEXE ']'
+ rm -rf /tmp/arm-builder.wINEuGvEXE
+ '[' -n /tmp/arm-builder.hv55mE6U0a ']'
+ rm -rf /tmp/arm-builder.hv55mE6U0a
+ '[' -n /tmp/arm-builder.jJBR9rn5tL ']'
+ umount /tmp/arm-builder.jJBR9rn5tL
umount: /tmp/arm-builder.jJBR9rn5tL: not mounted.
+ true
+ umount /dev/loop2
umount: /dev/loop2: not mounted.
+ true
+ rm -rf /tmp/arm-builder.jJBR9rn5tL
+ '[' -n /tmp/arm-builder.mrn9t3F16W ']'
+ rm -rf /tmp/arm-builder.mrn9t3F16W
+ '[' -n '' ']'
+ '[' -n '' ']'
+ '[' -n '' ']'
+ '[' -n '' ']'
+ '[' -n '' ']'
+ losetup -D /dev/loop2
+ dmsetup remove KairosVG-oem
device-mapper: remove ioctl on KairosVG-oem  failed: No such device or address
Command failed.
+ true
+ dmsetup remove KairosVG-recovery
device-mapper: remove ioctl on KairosVG-recovery  failed: No such device or address
Command failed.
+ true

Here my cloud-init file :

#cloud-config

install:
  auto: true
  device: "auto"
  extra-partitions:
    - name: storage
      label: storage
      size: 0
      fs: ext4

  reboot: true
# bundles:
  # This bundle needs to run after-install as it consumes assets from the LiveCD
  # which is not accessible otherwise at the first boot (there is no live-cd with any bundle.tar)
 #- targets:
 #  - run:///run/initramfs/live/bundle.tar
 #  local_file: true

# Define the user accounts on the node.
users:
  - name: "kairos"                       # The username for the user.
    passwd: "kairos"                      # The password for the user.
    ssh_authorized_keys:                  # A list of SSH keys to add to the user's authorized keys.
      - 'ssh-rsa remove_key             # A key from the user's GitHub account.
ssh_pwauth: False  ## This line enables ssh password authentication
k3s:
  enabled: true

Someone can help me to understand how I can generate a custom image to RPI4.

[jimmykarily]

rpi4 automatic partition expansion will be fixed in the upcoming 2.4.0 release. The relevant PR: kairos-io/packages#398

You build may be failing because of garbage devices left behind from previous builds or you may needs to reboot or run partprobe or something.

[fafarun]

Thanks for your response.
I need to wait the 2.4 to extend a partition?
Finally with this new build, I build my image with a large partition.

docker run -v $PWD:/HERE \
 -v $PWD/tmp:/tmp \
 -v /var/run/docker.sock:/var/run/docker.sock \
 --privileged -i --rm \
 --entrypoint=/build-arm-image.sh quay.io/kairos/osbuilder-tools:v0.8.6 \
 --use-lvm \
 --model rpi4 \
 --state-partition-size 26200 \
 --recovery-partition-size 4200 \
 --size 35000 \
 --images-size 30000 \
 --local \
 --config /HERE/cloud-config.yaml \
 --docker-image $IMAGE /HERE/build/out.img

But the overlay /var is steel with 2go. How can I manage this partition ?

The cloud-init with extend partition still not working :

install:
  auto: true
  device: "auto"
  extra-partitions:
    - name: storage
      label: storage
      size: 0
      fs: ext4

[jimmykarily]

As far as I can tell, your first example should have worked and the persistent partition should have expanded. This happens because of this file: https://github.com/kairos-io/packages/blob/main/packages/bundles/kairos-overlay-files/files/system/oem/11_persistent.yaml

Maybe this condition is not true on your device for some reason:

grep -i "Raspberry Pi 4" "/sys/firmware/devicetree/base/model"

You can run it manually and see.

The indentation is broken in your config (first test):

k3s:
enabled: true
args:
- --disable=traefik,servicelb

I assume it was a copy-paste error and it was correct when you created the image. Otherwise it's a problem that has to be fixed before we look further.

[fafarun]

I check my cloud-init, it's a copy-paste error. I fixed in my previous post to be clear.

With my first test image.

metal-6486:~ # grep -i "Raspberry Pi 4" "/sys/firmware/devicetree/base/model"
Binary file /sys/firmware/devicetree/base/model matches
metal-6486:~ # cat /sys/firmware/devicetree/base/model
Raspberry Pi 4 Model B Rev 1.4metal-6486:~ #

[jimmykarily]

This means the condition passes and kairos-agent should have tried to expand. Can you check sudo journalctl for errors? I'm sorry I can't help more. Expansion works on my RasberryPI and I'm not seeing something you are doing wrong so I have no way to reproduce.

[jimmykarily]

Also try to set rd.immucore.debug in the kernel cmdline (by hitting e when the grub menu shows up) and check the boot logs for errors. I'm not sure if those are accessible after booting, I usually use a serial cable and I get access to the logs on serial. Give a try though.

https://github.com/kairos-io/immucore/#kernel-configuration-parameters

[jimmykarily]

btw, it's a rasberry pi, the images come pre-installed. Thus you don't need an install: block.

[fafarun]

I tried with the new version 2.4, but the partition doesn't change. May be, I don't have the good configuration to do this.
Someone can publish your cloud-init and another configuration to see where is my problem.

[jimmykarily]

I gave it a try again with this image: quay.io/kairos/kairos-opensuse-leap-arm-rpi-img:v2.4.0-k3sv1.25.11-k3s1

and a config similar to yours above (including the otherwise not needed install block) and after booting the persistent partition wasn't expanded. I run kairos-agent run-stage rootfs.after manually and that returned an error about a non recognized filesystem for the mmc-something partition (I don't have the error message at hand, I will try again). Then I run the stage again manually and it complained that there is not enough unpartitioned space to expand. I checked and the partition was expanded (probably from the first manual run, even though it printed an error in the end).

So something is fishy but I need we need to try more things. At least, I can reproduce.

[jimmykarily]

I enabled debug output in immucore:

 [   87.251726] immucore[604]: time="2023-08-16T12:00:54Z" level=info msg="Extending last partition to max space"
 [   87.291589] immucore[604]: time="2023-08-16T12:00:54Z" level=error msg="FAILED: sfdisk not found\n"
 [   87.330331] immucore[604]: time="2023-08-16T12:00:54Z" level=error msg="failed to run growpart /dev/mmcblk0 5: exit status 2"
 [   87.370491] immucore[604]: time="2023-08-16T12:00:54Z" level=debug msg="Reading '/system/oem/00_rootfs.yaml'"
 [   87.410428] immucore[604]: time="2023-08-16T12:00:54Z" level=debug msg="Executing stage 'Rootfs Layout Settings.Grow persistent'"

[jimmykarily]

so sfdisk is missing from the initramfs, but it's there in the booted system. That's why running the stage manually after booting, expands the partition. Still, I'm getting a warning, but the partition seems expanded:

localhost:/home/kairos # kairos-agent run-stage rootfs.after
INFO[2023-09-18T09:47:35Z] kairos-agent version v2.2.11                 
INFO[2023-09-18T09:47:35Z] Running stage: rootfs.after.before           
INFO[2023-09-18T09:47:35Z] Done executing stage 'rootfs.after.before'   
INFO[2023-09-18T09:47:35Z] Running stage: rootfs.after                  
WARN[2023-09-18T09:47:35Z] (conditional) Skip 'Skipping stage (if statement error: failed to run [ -r /run/cos/custom-layout.env ] && [ ! -f "/run/cos/recovery_mode" ] && [ ! -f /run/cos/live_mode ]: exit status 1)' stage  
INFO[2023-09-18T09:47:35Z] Processing stage step 'Grow persistent'. ( commands: 0, files: 0, ... ) 
INFO[2023-09-18T09:47:35Z] Extending last partition to max space        
ERRO[2023-09-18T09:47:36Z]                                              
ERRO[2023-09-18T09:47:36Z] Could not find filesystem for /dev/mmcblk05, not resizing the filesystem 
INFO[2023-09-18T09:47:36Z] Processing stage step 'Grow persistent'. ( commands: 0, files: 0, ... ) 
WARN[2023-09-18T09:47:36Z] Not enough unpartitioned space in disk to operate 
INFO[2023-09-18T09:47:36Z] Done executing stage 'rootfs.after'          
INFO[2023-09-18T09:47:36Z] Running stage: rootfs.after.after            
INFO[2023-09-18T09:47:36Z] Done executing stage 'rootfs.after.after'    
INFO[2023-09-18T09:47:36Z] Running stage: rootfs.after.before           
INFO[2023-09-18T09:47:36Z] Done executing stage 'rootfs.after.before'   
INFO[2023-09-18T09:47:36Z] Running stage: rootfs.after                  
INFO[2023-09-18T09:47:36Z] Done executing stage 'rootfs.after'          
INFO[2023-09-18T09:47:36Z] Running stage: rootfs.after.after            
INFO[2023-09-18T09:47:36Z] Done executing stage 'rootfs.after.after'    
INFO[2023-09-18T09:47:36Z] Some errors found but were ignored. Enable --strict mode to fail on those or --debug to see them in the log 
WARN[2023-09-18T09:47:36Z] 1 error occurred:
        * Could not find filesystem for /dev/mmcblk05, not resizing the filesystem
 
localhost:/home/kairos # lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0     7:0    0    2G  1 loop /
mmcblk0 179:0    0 59.7G  0 disk 
├─mmcblk0p1
│       179:1    0   96M  0 part 
├─mmcblk0p2
│       179:2    0  6.1G  0 part /run/initramfs/cos-state
├─mmcblk0p3
│       179:3    0  4.1G  0 part 
├─mmcblk0p4
│       179:4    0   64M  0 part /oem
└─mmcblk0p5
        179:5    0 49.4G  0 part /usr/local/.state/usr-share-pki-trust.bind/anchors
                                 /usr/share/pki/trust/anchors
                                 /usr/share/pki/trust
                                 /var/lib/wicked
                                 /var/lib/snapd
                                 /var/lib/rancher
                                 /var/lib/longhorn
                                 /var/lib/kubelet
                                 /var/lib/dbus
                                 /var/lib/containerd
                                 /var/lib/cni
                                 /var/lib/ca-certificates
                                 /etc/zfs
                                 /etc/systemd
                                 /etc/sysconfig
                                 /etc/ssh
                                 /var/snap
                                 /etc/runlevels
                                 /etc/rancher
                                 /etc/modprobe.d
                                 /var/log
                                 /usr/libexec
                                 /etc/kubernetes
                                 /etc/iscsi
                                 /etc/cni
                                 /root
                                 /opt
                                 /home
                                 /usr/local

[jimmykarily]

This code: https://github.com/mudler/yip/blob/cc843a8f583b152f0891c687df3b3cb74e411589/pkg/plugins/layout.go#L464
works for sda1 but not for mmcblk0p5 . That's why we get the error about /dev/mmcblk05 (missing a p, the actual partition is /dev/mmcblk0p5).

So there were 2 bugs in there. @fafarun I'm sure I saw that working before we releasing 😕 , but I don't know how that is possible. Maybe sfdisk is there in some flavors? Or, more likely, maybe sfdisk is not a dependency of growpart in all distributions?

In any case, thanks for finding this and working on debugging this. I'll convert this to an issue and work on a fix.

[jimmykarily]

#1831 Let's continue there. Closing this.