arm32v7 failing to build

[tianon]

When building on arm32v7, the official servers are running into build failures -- here's a snippet of the relevant logs:

+ dpkg --print-architecture
+ dpkgArch=armhf
+ install_addin space https://github.com/JanWielemaker/space.git 8ab230a67e2babb3e81fac043512a7de7f4593bf
+ git clone https://github.com/JanWielemaker/space.git space
Cloning into 'space'...
+ git -C space checkout -q 8ab230a67e2babb3e81fac043512a7de7f4593bf
+ [ space = prosqlite ]
+ swipl -g pack_rebuild(space) -t halt
Warning: No build plugin could execute build step distclean
% checking for g++... 
% g++
% checking whether the C++ compiler works... 
% yes
% checking for C++ compiler default output file name... 
% a.out
% checking for suffix of executables... 
% 
% checking whether we are cross compiling... 
% no
% checking for suffix of object files... 
% o
% checking whether we are using the GNU C++ compiler... 
% yes
% checking whether g++ accepts -g... 
% yes
% checking for main in -lgeos... 
% yes
% checking for main in -lspatialindex... 
% yes
% checking how to run the C++ preprocessor... 
% g++ -E
% checking for grep that handles long lines and -e... 
% /bin/grep
% checking for egrep... 
% /bin/grep -E
% checking for ANSI C header files... 
% yes
% checking for sys/types.h... 
% yes
% checking for sys/stat.h... 
% yes
% checking for stdlib.h... 
% yes
% checking for string.h... 
% yes
% checking for memory.h... 
% yes
% checking for strings.h... 
% yes
% checking for inttypes.h... 
% yes
% checking for stdint.h... 
% yes
% checking for unistd.h... 
% yes
% checking spatialindex/SpatialIndex.h usability... 
% yes
% checking spatialindex/SpatialIndex.h presence... 
% yes
% checking for spatialindex/SpatialIndex.h... yes
% checking geos_c.h usability... 
% yes
% checking geos_c.h presence... 
% yes
% checking for geos_c.h... 
% yes
% checking for bzero... 
% yes
% configure: creating ./config.status
% config.status: creating Makefile
% config.status: creating config.h
% g++  -fPIC -pthread -I"/usr/lib/swipl/include" -I. -std=c++17 -c -o c/space.o c/space.cc
ERROR: In file included from c/globals.h:42,
ERROR:                  from c/space.cc:36:
ERROR: /usr/lib/swipl/include/SWI-cpp.h:165:3: error: ‘PlTerm::operator uint32_t() const’ cannot be overloaded with ‘PlTerm::operator term_t() const’
ERROR:   165 |   operator uint32_t(void) const;
ERROR:       |   ^~~~~~~~
ERROR: /usr/lib/swipl/include/SWI-cpp.h:158:3: note: previous declaration ‘PlTerm::operator term_t() const’
ERROR:   158 |   operator term_t(void) const
ERROR:       |   ^~~~~~~~
ERROR: /usr/lib/swipl/include/SWI-cpp.h:786:10: error: redefinition of ‘PlTerm::operator uint32_t() const’
ERROR:   786 | __inline PlTerm::operator uint32_t(void) const
ERROR:       |          ^~~~~~
ERROR: /usr/lib/swipl/include/SWI-cpp.h:158:3: note: ‘PlTerm::operator term_t() const’ previously defined here
ERROR:   158 |   operator term_t(void) const
ERROR:       |   ^~~~~~~~
Warning: c/space.cc: In function ‘foreign_t pl_rtree_set_space__2(PlTermv)’:
Warning: c/space.cc:144:36: warning: ISO C++ says that these are ambiguous, even though the worst conversion for the first is better than the worst conversion for the second:
Warning:   144 |     idx->utilization = (double)A2[1];
Warning:       |                                    ^
Warning: In file included from c/globals.h:42,
Warning:                  from c/space.cc:36:
Warning: /usr/lib/swipl/include/SWI-cpp.h:834:1: note: candidate 1: ‘PlTerm PlTerm::operator[](size_t) const’
Warning:   834 | PlTerm::operator [](ARITY_T index) const
Warning:       | ^~~~~~
Warning: c/space.cc:144:36: note: candidate 2: ‘operator[](char*, int)’ (built-in)
Warning:   144 |     idx->utilization = (double)A2[1];
Warning:       |                                    ^
Warning: c/space.cc:144:36: warning: ISO C++ says that these are ambiguous, even though the worst conversion for the first is better than the worst conversion for the second:
Warning: In file included from c/globals.h:42,
Warning:                  from c/space.cc:36:
Warning: /usr/lib/swipl/include/SWI-cpp.h:834:1: note: candidate 1: ‘PlTerm PlTerm::operator[](size_t) const’
Warning:   834 | PlTerm::operator [](ARITY_T index) const
Warning:       | ^~~~~~
Warning: c/space.cc:144:36: note: candidate 2: ‘operator[](wchar_t*, int)’ (built-in)
Warning:   144 |     idx->utilization = (double)A2[1];
Warning:       |                                    ^
Warning: c/space.cc:146:30: warning: ISO C++ says that these are ambiguous, even though the worst conversion for the first is better than the worst conversion for the second:
Warning:   146 |     idx->nodesize = (int)A2[1];
Warning:       |                              ^
Warning: In file included from c/globals.h:42,
Warning:                  from c/space.cc:36:
Warning: /usr/lib/swipl/include/SWI-cpp.h:834:1: note: candidate 1: ‘PlTerm PlTerm::operator[](size_t) const’
Warning:   834 | PlTerm::operator [](ARITY_T index) const
Warning:       | ^~~~~~
Warning: c/space.cc:146:30: note: candidate 2: ‘operator[](char*, int)’ (built-in)
Warning:   146 |     idx->nodesize = (int)A2[1];
Warning:       |                              ^
Warning: c/space.cc:146:30: warning: ISO C++ says that these are ambiguous, even though the worst conversion for the first is better than the worst conversion for the second:
Warning: In file included from c/globals.h:42,
Warning:                  from c/space.cc:36:
Warning: /usr/lib/swipl/include/SWI-cpp.h:834:1: note: candidate 1: ‘PlTerm PlTerm::operator[](size_t) const’
Warning:   834 | PlTerm::operator [](ARITY_T index) const
Warning:       | ^~~~~~
Warning: c/space.cc:146:30: note: candidate 2: ‘operator[](wchar_t*, int)’ (built-in)
Warning:   146 |     idx->nodesize = (int)A2[1];
Warning:       |                              ^
Warning: c/space.cc:148:13: warning: ISO C++ says that these are ambiguous, even though the worst conversion for the first is better than the worst conversion for the second:
Warning:   148 |     if (A2[1].name() == ATOM_memory) {
Warning:       |             ^
Warning: In file included from c/globals.h:42,
Warning:                  from c/space.cc:36:
Warning: /usr/lib/swipl/include/SWI-cpp.h:834:1: note: candidate 1: ‘PlTerm PlTerm::operator[](size_t) const’
Warning:   834 | PlTerm::operator [](ARITY_T index) const
Warning:       | ^~~~~~
Warning: c/space.cc:148:13: note: candidate 2: ‘operator[](char*, int)’ (built-in)
Warning:   148 |     if (A2[1].name() == ATOM_memory) {
Warning:       |             ^
Warning: c/space.cc:148:13: warning: ISO C++ says that these are ambiguous, even though the worst conversion for the first is better than the worst conversion for the second:
Warning: In file included from c/globals.h:42,
Warning:                  from c/space.cc:36:
Warning: /usr/lib/swipl/include/SWI-cpp.h:834:1: note: candidate 1: ‘PlTerm PlTerm::operator[](size_t) const’
Warning:   834 | PlTerm::operator [](ARITY_T index) const
Warning:       | ^~~~~~
Warning: c/space.cc:148:13: note: candidate 2: ‘operator[](wchar_t*, int)’ (built-in)
Warning:   148 |     if (A2[1].name() == ATOM_memory) {
Warning:       |             ^
Warning: c/space.cc:150:20: warning: ISO C++ says that these are ambiguous, even though the worst conversion for the first is better than the worst conversion for the second:
Warning:   150 |     } else if (A2[1].name() == ATOM_disk) {
Warning:       |                    ^
Warning: In file included from c/globals.h:42,
Warning:                  from c/space.cc:36:
Warning: /usr/lib/swipl/include/SWI-cpp.h:834:1: note: candidate 1: ‘PlTerm PlTerm::operator[](size_t) const’
Warning:   834 | PlTerm::operator [](ARITY_T index) const
Warning:       | ^~~~~~
Warning: c/space.cc:150:20: note: candidate 2: ‘operator[](char*, int)’ (built-in)
Warning:   150 |     } else if (A2[1].name() == ATOM_disk) {
Warning:       |                    ^
Warning: c/space.cc:150:20: warning: ISO C++ says that these are ambiguous, even though the worst conversion for the first is better than the worst conversion for the second:
Warning: In file included from c/globals.h:42,
Warning:                  from c/space.cc:36:
Warning: /usr/lib/swipl/include/SWI-cpp.h:834:1: note: candid
Warning: ate 1: ‘PlTerm PlTerm::operator[](size_t) const’
Warning:   834 | PlTerm::operator [](ARITY_T index) const
Warning:       | ^~~~~~
Warning: c/space.cc:150:20: note: candidate 2: ‘operator[](wchar_t*, int)’ (built-in)
Warning:   150 |     } else if (A2[1].name() == ATOM_disk) {
Warning:       |                    ^
Warning: c/space.cc:157:13: warning: ISO C++ says that these are ambiguous, even though the worst conversion for the first is better than the worst conversion for the second:
Warning:   157 |     if (A2[1].name() == ATOM_pythagorean) {
Warning:       |             ^
Warning: In file included from c/globals.h:42,
Warning:                  from c/space.cc:36:
Warning: /usr/lib/swipl/include/SWI-cpp.h:834:1: note: candidate 1: ‘PlTerm PlTerm::operator[](size_t) const’
Warning:   834 | PlTerm::operator [](ARITY_T index) const
Warning:       | ^~~~~~
Warning: c/space.cc:157:13: note: candidate 2: ‘operator[](char*, int)’ (built-in)
Warning:   157 |     if (A2[1].name() == ATOM_pythagorean) {
Warning:       |             ^
Warning: c/space.cc:157:13: warning: ISO C++ says that these are ambiguous, even though the worst conversion for the first is better than the worst conversion for the second:
Warning: In file included from c/globals.h:42,
Warning:                  from c/space.cc:36:
Warning: /usr/lib/swipl/include/SWI-cpp.h:834:1: note: candidate 1: ‘PlTerm PlTerm::operator[](size_t) const’
Warning:   834 | PlTerm::operator [](ARITY_T index) const
Warning:       | ^~~~~~
Warning: c/space.cc:157:13: note: candidate 2: ‘operator[](wchar_t*, int)’ (built-in)
Warning:   157 |     if (A2[1].name() == ATOM_pythagorean) {
Warning:       |             ^
Warning: c/space.cc:159:20: warning: ISO C++ says that these are ambiguous, even though the worst conversion for the first is better than the worst conversion for the second:
Warning:   159 |     } else if (A2[1].name() == ATOM_haversine) {
Warning:       |                    ^
Warning: In file included from c/globals.h:42,
Warning:                  from c/space.cc:36:
Warning: /usr/lib/swipl/include/SWI-cpp.h:834:1: note: candidate 1: ‘PlTerm PlTerm::operator[](size_t) const’
Warning:   834 | PlTerm::operator [](ARITY_T index) const
Warning:       | ^~~~~~
Warning: c/space.cc:159:20: note: candidate 2: ‘operator[](char*, int)’ (built-in)
Warning:   159 |     } else if (A2[1].name() == ATOM_haversine) {
Warning:       |                    ^
Warning: c/space.cc:159:20: warning: ISO C++ says that these are ambiguous, even though the worst conversion for the first is better than the worst conversion for the second:
Warning: In file included from c/globals.h:42,
Warning:                  from c/space.cc:36:
Warning: /usr/lib/swipl/include/SWI-cpp.h:834:1: note: candidate 1: ‘PlTerm PlTerm::operator[](size_t) const’
Warning:   834 | PlTerm::operator [](ARITY_T index) const
Warning:       | ^~~~~~
Warning: c/space.cc:159:20: note: candidate 2: ‘operator[](wchar_t*, int)’ (built-in)
Warning:   159 |     } else if (A2[1].name() == ATOM_haversine) {
Warning:       |                    ^
% gmake: *** [Makefile:23: c/space.o] Error 1
ERROR: -g pack_rebuild(space): Process "process(/usr/bin/gmake,[])": exit status: 2

[JanWielemaker]

Looks like the same issues with the C++ interface that bother rocksdb an hdt. Work is going on to resolve these issues using a better designed C++ interface. In the meanwhile I propose to disable space in the same way.

[tianon]

Looks like it's hdt causing problems now: 👀

mkdir -p lib/armv8l-linux
g++   -shared -o lib/armv8l-linux/hdt4pl.so c/hdt4pl.o -Lc -lhdt -lcds  -lserd-0
gmake: Nothing to be done for 'install'.
/usr/lib/swipl/bin/armv8l-linux/swipl -g test_hdt -t halt test/test_hdt.pl
% Start unit: hdt
% [1/1] hdt:hdt_search_id ..Reading: 4 Read: 767
....................... **FAILED (0.010 sec)
ERROR: /usr/share/swi-prolog/pack/hdt/test/test_hdt.pl:20:
ERROR: ERROR:     test hdt:hdt_search_id: received error: HDT: Could not read the expected bytes from input
% End unit hdt: **FAILED (0.013 sec CPU)
ERROR: 1 test failed
% 0 tests passed
% % Test run completed in 0.020 seconds (0.020 cpu)
ERROR: -g test_hdt: false
gmake: *** [Makefile:42: check] Error 1
ERROR: -g pack_rebuild(hdt): Process "process(/usr/bin/gmake,[check])": exit status: 2

[JanWielemaker]

Which Prolog version and version of the pack are we talking about?

[kamahen]

Has the first problem (compilation errors in pack space) gone away and there's a new problem, namely the test failure in pack hdt?
Can I assume that the build+test is done from a clean image?
Is this a problem only with arm32v7 (or armv8l?) or does it show in other images?
Which version of SWI-Prolog and version of hdt? (that is, which commit in git@github.com:JanWielemaker/hdt.git and git@github.com:SWI-Prolog/swipl-devel.git)
And we might need to know the versions of libserd-dev or serd-devel because this appears to be an error message from one of those, that is being reported via the Prolog test.

[tianon]

Both latest and stable are currently failing (docker buildx build --platform linux/arm/v7 https://github.com/SWI-Prolog/docker-swipl.git#4c15bb03b82ab39fb3bdf6579ad66014111faa11:9.3.9/bookworm and docker buildx build --platform linux/arm/v7 https://github.com/SWI-Prolog/docker-swipl.git#4c15bb03b82ab39fb3bdf6579ad66014111faa11:9.2.6/bookworm).

So that's:

docker-swipl/9.3.9/bookworm/Dockerfile

Line 72 in 4c15bb0

install_addin hdt https://github.com/JanWielemaker/hdt.git 7f2221747ea751a20ad0d7b95aebfd2c99649c1f; \

and

docker-swipl/9.2.6/bookworm/Dockerfile

Line 72 in 4c15bb0

install_addin hdt https://github.com/JanWielemaker/hdt.git 7f2221747ea751a20ad0d7b95aebfd2c99649c1f; \

(as you can see in that Dockerfile, the previous issues with space don't currently apply because the line that installs space is commented out completely, but even if it weren't, it excludes armhf / arm32v7 explicitly 😅)

[kamahen]

This is not the latest version of hdt, so changing to commit 18c3a52fe4d094f8f19a9571896d9a8273e44fe7 might fix things.
The space commit seems to be the latest, so that might fix the original reported problem (from over a year ago); perhaps uncomment line 72?

(The decision on these needs to be made by @JanWielemaker -- I merely updated the packs to use the new C++ API)

[tianon]

I ended up going slightly newer in the history on hdt (JanWielemaker/hdt@f6a5f74) because the commits after JanWielemaker/hdt@18c3a52 seemed pretty sane:

diff --git a/9.3.9/bookworm/Dockerfile b/9.3.9/bookworm/Dockerfile
index 755954a..3a44b32 100644
--- a/9.3.9/bookworm/Dockerfile
+++ b/9.3.9/bookworm/Dockerfile
@@ -69,7 +69,7 @@ RUN set -eux; \
     # libgeos-dev of Debian bookworm is broken: the library does not define the Coordinate constructors nor the inline code to avoid calling these.
     # [ "$dpkgArch" = 'armhf' ] || [ "$dpkgArch" = 'armel' ] || install_addin space https://github.com/JanWielemaker/space.git 097687c7b67902704b03f095cc2f1e9.3.91e700; \
     [ "$dpkgArch" = 'armhf' ] || [ "$dpkgArch" = 'armel' ] || install_addin rocksdb https://github.com/JanWielemaker/rocksdb.git a63f1f5650e44c7d40401ed5a8b689aa1caca635; \
-    install_addin hdt https://github.com/JanWielemaker/hdt.git 7f2221747ea751a20ad0d7b95aebfd2c99649c1f; \
+    install_addin hdt https://github.com/JanWielemaker/hdt.git f6a5f746949ab5a150cce4e47b45049cefd18e8a; \
     [ "$dpkgArch" = 'armhf' ] || [ "$dpkgArch" = 'armel' ] || install_addin rserve_client https://github.com/JanWielemaker/rserve_client.git bdf8962264d65dd8ef6eedf5f00ff0c0f6c52c2f; \
     apt-get purge -y --auto-remove $BUILD_DEPS
 CMD ["swipl"]

This builds fine on amd64 and arm64, but on arm32 (v7), it still fails like this:

/usr/bin/gmake  check-TESTS
gmake[4]: Entering directory '/usr/share/swi-prolog/pack/hdt/hdt-cpp/libhdt/tests'
gmake[5]: Entering directory '/usr/share/swi-prolog/pack/hdt/hdt-cpp/libhdt/tests'
PASS: c11
PASS: bitutiltest
PASS: listener
PASS: streamtest
PASS: bit375
% PASS: testmax
PASS: properties
% PASS: serd
PASS: logarr
../../build/test-driver: line 112: 69560 Aborted                 (core dumped) "$@" >> "$log_file" 2>&1
FAIL: dumpDictionary
FAIL: dictionaries
../../build/test-driver: line 112: 69569 Aborted                 (core dumped) "$@" >> "$log_file" 2>&1
============================================================================
% Testsuite summary for libhdt 1.3.3
% ============================================================================
% # TOTAL: 11
% # PASS:  9
# SKIP:  0
% # XFAIL: 0
% # FAIL:  2
% # XPASS: 0
% # ERROR: 0
% ============================================================================
See libhdt/tests/test-suite.log
% Please report to some@email.com
% ============================================================================
gmake[5]: Leaving directory '/usr/share/swi-prolog/pack/hdt/hdt-cpp/libhdt/tests'
gmake[5]: *** [Makefile:822: test-suite.log] Error 1
gmake[4]: *** [Makefile:930: check-TESTS] Error 2
gmake[4]: Leaving directory '/usr/share/swi-prolog/pack/hdt/hdt-cpp/libhdt/tests'
gmake[3]: *** [Makefile:1073: check-am] Error 2
gmake[3]: Leaving directory '/usr/share/swi-prolog/pack/hdt/hdt-cpp/libhdt/tests'
gmake[2]: Leaving directory '/usr/share/swi-prolog/pack/hdt/hdt-cpp/libhdt'
gmake[2]: *** [Makefile:1656: check-recursive] Error 1
gmake[1]: Leaving directory '/usr/share/swi-prolog/pack/hdt/hdt-cpp'
gmake[1]: *** [Makefile:432: check-recursive] Error 1
% gmake: *** [Makefile:49: check] Error 2
ERROR: -g pack_rebuild(hdt): Process "process(/usr/bin/gmake,[check])": exit status: 2

However, this does look healthier than the error we were getting before, so it's at least possible this is QEMU user-mode emulation related and might be worth trying on actual hardware.

[tianon]

Unfortunately the failure looks the same on native Arm hardware:

/usr/bin/gmake  check-TESTS
gmake[4]: Entering directory '/usr/share/swi-prolog/pack/hdt/hdt-cpp/libhdt/tests'
gmake[5]: Entering directory '/usr/share/swi-prolog/pack/hdt/hdt-cpp/libhdt/tests'
PASS: listener
PASS: bit375
PASS: c11
PASS: serd
PASS: streamtest
PASS: properties
PASS: logarr
../../build/test-driver: line 112: 26485 Aborted                 (core dumped) "$@" >> "$log_file" 2>&1
FAIL: dictionaries
PASS: testmax
PASS: bitutiltest
../../build/test-driver: line 112: 26606 Aborted                 (core dumped) "$@" >> "$log_file" 2>&1
FAIL: dumpDictionary
============================================================================
% Testsuite summary for libhdt 1.3.3
% ============================================================================
# TOTAL: 11
% # PASS:  9
% # SKIP:  0
# XFAIL: 0
# FAIL:  2
# XPASS: 0
# ERROR: 0
% ============================================================================
% See libhdt/tests/test-suite.log
Please report to some@email.com
% ============================================================================
gmake[5]: *** [Makefile:822: test-suite.log] Error 1
gmake[5]: Leaving directory '/usr/share/swi-prolog/pack/hdt/hdt-cpp/libhdt/tests'
gmake[4]: *** [Makefile:930: check-TESTS] Error 2
gmake[4]: Leaving directory '/usr/share/swi-prolog/pack/hdt/hdt-cpp/libhdt/tests'
gmake[3]: *** [Makefile:1073: check-am] Error 2
gmake[3]: Leaving directory '/usr/share/swi-prolog/pack/hdt/hdt-cpp/libhdt/tests'
gmake[2]: *** [Makefile:1656: check-recursive] Error 1
gmake[2]: Leaving directory '/usr/share/swi-prolog/pack/hdt/hdt-cpp/libhdt'
gmake[1]: *** [Makefile:432: check-recursive] Error 1
gmake[1]: Leaving directory '/usr/share/swi-prolog/pack/hdt/hdt-cpp'
gmake: *** [Makefile:49: check] Error 2
ERROR: -g pack_rebuild(hdt): Process "process(/usr/bin/gmake,[check])": exit status: 2

[kamahen]

Looking at hdt, I see that the official package (at https://www.swi-prolog.org/pack/list?p=hdt) uses the release from 2023-12-18; the docker image uses 2024-07-12; and the last update in GitHub was 2024-07-14. So, it's possible that the problem has been fixed in the interim (for example, there has been an update to the submodule hdt-cpp).

I still have a few more small-ish changes to make to the hdt package -- I probably won't be able to do them for another week or two ... I'm hoping that this is OK.

In the meantime, is there any way for me to try things out on an ARM machine (or emulator)? And is it only the 32-bit ARM builds that have the problem? (If so, should we even support 32-bit?)

(I think that there are similar version mis-matches for packages space, rserver_client and rocksdb ... I intend to do my final small changes to them at the same time as the hdt package, and then @JanWielemaker can set the packages and docker images to the latest commits.)

[tianon]

In the meantime, is there any way for me to try things out on an ARM machine (or emulator)?

Yes! I only tried on real hardware because segfault is one of those behaviors that is often the result of not emulating the architecture correctly, but the issue reproduces the same on real hardware, so emulation should be fine to reproduce. 😄

I use the qemu-user-static package installed on my (Debian) host, which uses the kernel's binfmt support to install the user-mode emulator such that I can then transparently run "foreign" architecture Docker containers without issue. Docker Desktop (on all platforms) does a similar setup by default IIRC. My experience is that this is usually good enough for testing things, but be warned it does often have quirks that can only be overcome by doing full system-mode emulation (ie, full emulated VM) or running on native hardware. 😅

And is it only the 32-bit ARM builds that have the problem? (If so, should we even support 32-bit?)

Yeah, as far as I can tell it's only 32bit Arm builds -- the primary audience for these would be anyone on a Raspberry Pi (or similar hardware from other vendors), although we're now down to just v7 so only Raspberry Pi 2+ (no Pi1 or Pi0), and Pi3+ is technically an arm64 system so users can/arguably should be running a 64bit OS on those.

In general, the audience for 32bit Arm does seem to be shrinking, but slowly, and they tend to be vocal to an outsized degree. However, that being said, I don't know whether there's much demand for the swipl image specifically on these platforms (as evidenced by the lack of anyone but me and y'all even reacting to this issue/thread 😄).

[kamahen]

At least one of the error messages appears (at first glance) to be from the underlying code that is merely wrapped by SWI-Prolog, so fixing that might be beyond my capabilities.

I would vote for simply disabling the three packages for 32-bit Raspberry Pi, as unlikely to be used there; and if someone does want one of those packages, there's a straightforward mechanism for installing it (although it might not work, of course).
@JanWielemaker - what do you think? Is there something special about HDT, space, rserver_client, that only they are included in the images and not any of the other packages?