Building working images

[dirkhh]

I'm certain that I just missed a piece of information that's hiding in plain sight.
When I try to build my own image it comes without kernel/devicetree.
This is on Ubuntu Jammy, latest git master for DietPi (17a3275 as I write this).
I create a build directory parallel to the DietPi directory and run sudo bash ../DietPi/.build/images/dietpi-build -m 52
But the resulting image has only the DietPi files in the /boot directory. No kernel, no devicetree, nothing.
Which makes me think that I am missing something.
Please feel free to tell me to RTFM - ideally pointing me at the relevant document 🙃

[Joulinar]

We don't support Ubuntu at all. You need to use Debian image.

Not sure if the build script is working outside GitHub because we use it for GitHub Actions. Furthermore it seem you did not specified architecture (-a) and Debian destro (-d).

Btw there is no documentation for this feature as it is not intended for public usage. Usually people use the DietPi install script to build their own image and not the build script. https://dietpi.com/docs/hardware/#make-your-own-distribution

[dirkhh]

I'm somewhat confused. The GitHub action seems to run on Ubuntu. I must be misunderstanding things.

The bigger goal of my project is to create DietPi derived images that I can distribute to users, so doing the conversion at runtime really isn't a workable approach for that.

Right now I'm using existing Armbian and DietPi images and CustomPios to add a few things. Being able to just build images using the existing scripts would be much easier. And fundamentally I don't see why something would work in an GitHub action and not when recreated in a VM.

BTW, I don't think the GitHub action explicitly sets all the arguments. But I guess I need to spend more time starting at those logs...

[Joulinar]

ok maybe we misunderstood. Our images are based on Debian at the end not Ubuntu. 😄

I guess it is needed to pass architecture (-a) and Debian destro (-d) to the build script. Otherwise, the script doesn't know which image to build. Basically, each SBC type has an own individual image. But @MichaIng knows it way better than I.

[MichaIng]

Yes the dietpi-build script works on Ubuntu as well, the GitHub runners themselves are Ubuntu VMs.

[MichaIng]

Interesting, it worked well here two weeks ago: https://github.com/MichaIng/DietPi/actions/runs/5718248624/job/15499287698
Relevant step is line 1678 where kernel, dtb and bootloader packages are installed. Probably you see differences?

I also did just trigger a build with current master branch and same options: https://github.com/MichaIng/DietPi/actions/runs/5842585596

-d 7 (Bookworm) is the default, so that should have the same result. Also the architecture does not need to be defined for ASUS TB, only for RPi and containers where we do support multiple architectures for the same hardware ID.

[dirkhh]

That's super helpful.
I'm trying to get the output into a logfile, but in my case I always get the interactive question about the "current device this is being installed on" - and when redirecting output I appear unable to interact with that. But also, clearly running on GitHub Actions you don't get that question, so it's even more obvious that there's something that I'm doing "differently" (which sounds nicer that "wrong")

[dirkhh]

actually, I managed to hack around this an here's where our logs diverge:
mine:

[FAILED] Failed to start dropbear.service - Lightweight SSH server.
See 'systemctl status dropbear.service' for details.
Reading package lists... Done
[ INFO ] Selected Git branch: MichaIng/master
[ INFO ] DietPi-Installer | Detected distribution version: Bookworm (ID: 7)
[ INFO ] DietPi-Installer | Detected target CPU architecture: aarch64 (ID: 3)

 DietPi-Installer
─────────────────────────────────────────────────────
 Step: [0] Detecting existing DietPi system

[ INFO ] DietPi-Installer | No DietPi system found, skipping old instance uninstall...

 DietPi-Installer
─────────────────────────────────────────────────────
 Step: [1] Target system inputs

[ INFO ] DietPi-Installer | Entered image creator: 0
[ INFO ] DietPi-Installer | Entered pre-image info: 0
[ INFO ] DietPi-Installer | Selected hardware model ID: 22
[ INFO ] DietPi-Installer | Marking WiFi as required
[ INFO ] DietPi-Installer | Selected Debian version: bookworm (ID: 7)

GitHub Action:

[FAILED] Failed to start dropbear.s…vice - Lightweight SSH server.
See 'systemctl status dropbear.service' for details.
removed '/etc/apt/apt.conf.d/01autoremove'
0% [Working]       
Get:1 https://deb.debian.org/debian bookworm InRelease [151 kB]
0% [1 InRelease 16.4 kB/151 kB 11%]                              
0% [Working]
0% [Working]
0% [Working]
0% [Working]
0% [Working]
0% [Working]
15% [Working]        
Get:2 https://deb.debian.org/debian bookworm/main armhf Packages [8612 kB]
15% [2 Packages 0 B/8612 kB 0%]
93% [Working]
93% [2 Packages store 0 B]
93% [2 Packages store 0 B]
93% [2 Packages store 0 B]
93% [2 Packages store 0 B]
93% [2 Packages store 0 B]
93% [2 Packages store 0 B]                                         1363 kB/s 0s
93% [2 Packages store 0 B]                                         1363 kB/s 0s
93% [2 Packages store 0 B]                                         1363 kB/s 0s
93% [2 Packages store 0 B]                                         1363 kB/s 0s
100% [Working]                                                     1363 kB/s 0s
Fetched 8763 kB in 8s (1066 kB/s)
Reading package lists... 0%
Reading package lists... 0%
Reading package lists... 0%
Reading package lists... 9%
Reading package lists... 18%
Reading package lists... 27%
Reading package lists... 36%
Reading package lists... 45%
Reading package lists... 54%
Reading package lists... 62%
Reading package lists... 71%
Reading package lists... 80%
Reading package lists... 88%
Reading package lists... 97%
Reading package lists... 99%
Reading package lists... 99%
Reading package lists... Done
[ INFO ] Selected Git branch: MichaIng/master
[ INFO ] DietPi-Installer | Detected distribution version: Bookworm (ID: 7)
[ INFO ] DietPi-Installer | Detected target CPU architecture: armv7l (ID: 2)

 DietPi-Installer
─────────────────────────────────────────────────────
 Step: [0] Detecting existing DietPi system
[ INFO ] DietPi-Installer | No DietPi system found, skipping old instance uninstall...

 DietPi-Installer
─────────────────────────────────────────────────────
 Step: [1] Target system inputs
[ INFO ] DietPi-Installer | Entered image creator: 0
[ INFO ] DietPi-Installer | Entered pre-image info: 0
[ INFO ] DietPi-Installer | Selected hardware model ID: 52
[ INFO ] DietPi-Installer | Marking WiFi as required
[ INFO ] DietPi-Installer | Selected Debian version: bookworm (ID: 7)

So there's clearly some update happening after the FAILED dropbear and before the installer starts. That's odd.
And then it simply accepts the choice of machine '52' for you, while for me it brings up an interactive dialog where '52' isn't an option (which is why I'm manually choosing 22 - the generic option). And I think that is then the reason why I'm not getting the kernel and dtb. Am I close?

So here's a totally wild question -- could this be because I'm running this on an arm64 system? So somehow the installer thinks that I'm running on device, instead of on an amd64 system which would indicate that it is being cross built???

[dirkhh]

yup - looks like that was it.
I spun up an amd64 VM and am running it on there. Runs SIGNIFICANTLY slower than on the arm64 one, but on the flip side it works.
So somewhere in the code there's an assumption that if the system that is executing the code is an arm one, then it isn't cross building.
Longer term it may be worth removing that assumption :)
For now I think I have what I need. Thank you so much for your help.
I wonder if it's ok to mark my own summary as the answer as that actually explains what happens... of course the credit for helping me figure it out should go to you...

[MichaIng]

Ah right, the arm64 system is the reason: It does not use QEMU for emulation then, as arm64 can run armv6/7 code natively (and the arm64 QEMU package does not even contain an armv7 emulator), so within the installer you get

[ INFO ] DietPi-Installer | Detected target CPU architecture: armv7l (ID: 2)

and the installer allows and shows only options matching the architecture, and ASUS TB has a ARMv7 CPU. The other way round, it shows ARMv8 capable options when detecting a ARMv7 CPU, as it is possible that those are running with a 32-bit kernel (and userland) where uname -m returns armv7l.

So basically the installer shows all possible hardware options, but assuming it is executed on the actual hardware rather then within a container on a backwards compatible machine. For x86_64 it does not matter, as it then requires emulation in any case but x86_64. But for ARM this is conflicting and we would need to allow and show any ARM option when detecting any ARM architecture 🤔.

I have an idea: A container can be actually detected via systemd-detect-virt -c. So if that returns true, we can add all ARM options, else assume native hardware.

[dirkhh]

That would be awesome.
It does run a lot faster on the arm64 system 😄

Did you want me to figure how and where to do this and submit a PR, or are you doing this already?

[dirkhh]

I submitted a PR - the code doesn't pass shellcheck - not sure how to do this right 🤷🏼‍♂️🤣
#6547