Noob question - Docker images from DockerHub not working correctly.

[pbisbal1]

Okay - I'm very green with Charliecloud - I've only been using it for the past 30 hours or so, and I've spent most of that time trying to get an application I pulled down from DockerHub with 'ch-image pull' to work as advertised. It's my understanding that if I pull an image with 'ch-image pull', Charliecloud puils down the image and builds it in it's own format, and it should be ready to run. Is that correct?

I've done some of the simple examples in the tutorial, and those images exhibit the same problems, so I think I must be the problem here. Here's what I'm seeing:

1. The environment variables specified in the Dockerfile are not propagated to the image
2. The command specified by CMD doesn't execute when I start the image, and I get an error message that I didn't specify an image or a command to the ch-run command.
3. When I specify all the environment variables and the command to execute on the command-line, I get errors about libraries not being found, which I've traced to the version(s) of that library found in that image not being the same as what the application was linked to.

As a sanity check, I've tried this with Docker, and with Charliecloud on my personal computer to see if what something with my work systems' environment or Linux version. Everything works as directed with Docker, and Charliecloud behaves the same on my personal system.

Am I missing something? Here's what I'm doing:

1. export CH_IMAGE_STORAGE=/path/to/container/storage
2. ch-image pull plasmapotential/heat:v3.0.1
3. ch-run plasmapotential/heat:v3.0.1

Thanks for the help,
Prentice

[reidpr]

Hello Prentice,

Welcome to Charliecloud!

• The environment variables specified in the Dockerfile are not propagated to the image

This is the default behavior, but if you specify ch-run --set-env it should set the environment as specified in the Dockerfile. You may in fact need ch-run --unset-env='*' --set-env, which should clear the environment before setting it as specified in the Dockerfile.

• The command specified by CMD doesn't execute when I start the image, and I get an error message that I didn't specify an image or a command to the ch-run command.

Yes, we don't currently support CMD. I believe there is an open bug to do so.

The reasoning was that Charliecloud is designed for HPC applications, and those applications lend themselves less well to being repeatedly run with the same argument (than the web apps containers were originally used for). However, yours isn't the only feedback we've received that such support would be useful.

• When I specify all the environment variables and the command to execute on the command-line, I get errors about libraries not being found, which I've traced to the version(s) of that library found in that image not being the same as what the application was linked to.

Specifics would be needed to debug this. However, there are a lot of environment variables that control linking, so I wonder if you've been caught up in that.

If you post the Dockerfile and the ldd output for your application for both Charliecloud, we can look into it.

HTH,
Reid

[pbisbal1]

Reid,

Thanks for the help! I was using --setenv to set all the environment variables manually. I didn't realize that using it w/o defining an environment variable tells it to use /ch/environment. I re-read that section of ch-run, and sure enough, it states that clear as day. Me culpa!

I understand your logic behind not supporting CMD for HPC applications, but I think it would be useful to those, so add me to the list of those who'd like to see it supported. It's easy enough to specify the argument on the command-line.

Using --set-env as you instructed helps a lot. Here's what happens when I try to start the app in the container, showing the linking error:

$ ch-run --home --set-env plasmapotential/heat:v3.0.1 -- python3 /root/source/HEAT/launchHEAT.py --a 0.0.0.0 --m g
Running in Docker mode

Running HEAT via Graphical User Interface (GUI)...

Traceback (most recent call last):
  File "/root/source/HEAT/launchHEAT.py", line 319, in <module>
    launchHEAT(args)
  File "/root/source/HEAT/launchHEAT.py", line 268, in launchHEAT
    import dashGUI as dashGUI
  File "/root/source/HEAT/dashGUI.py", line 82, in <module>
    from engineClass import engineObj
  File "/root/source/HEAT/engineClass.py", line 12, in <module>
    import CADClass
  File "/root/source/HEAT/CADClass.py", line 21, in <module>
    import FreeCAD
ImportError: libQt5Core.so.5: cannot open shared object file: No such file or directory

When I run a bash shell in the container, and then use ldd to look at the library and executables associated with FreeCAD, they are looking for libQtCore.so.5 version 5.15. For example:

$ ldd libFreeCADBase.so  | grep "not found"
	libQt5Core.so.5 => not found

Using "ldd -v" provids this additional information:

	libQt5Core.so.5 (Qt_5.15) => not found
	libQt5Core.so.5 (Qt_5) => not found

There are two different libQt5Core.so.5 files in the image:

$ find / -name libQt5Core.so.5 
/usr/lib/x86_64-linux-gnu/libQt5Core.so.5
/root/opt/paraview/lib/libQt5Core.so.5

/usr/lib/x86_64-linux-gnu should be part of the system-wide ld search path set in /etc/ld.so.conf.d/*, and it is:

$ more /etc/ld.so.conf.d/x86_64-linux-gnu.conf 
# Multiarch support
/usr/local/lib/x86_64-linux-gnu
/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu

And the Dockerfile isn't adding the paraview directory to the path due to conflicts between what Paraview wants and what other apps want:

#ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:/root/opt/paraview/lib #comment to prevent qt5 conflicts

[reidpr]

Is the Dockerfile something you can share in full? (or a similar one that also exhibits the undesired behavior)

[pbisbal1]

Reid,

I sure can. I pulled it off of DockerHub, so it's public:

#Dockerfile
#Engineer: T Looby
#Date: 01/27/2022
#Description:  Dockerfile for building HEAT docker image
#
#this script should only be run on dev machine
#script is not run directly, but is used when buildDocker bash
#script is run.  buildDocker script calls this file using syntax
#similar to this:
# docker build -t plasmapotential/heat -f ./Dockerfile ./github/source

# start from base
FROM ubuntu:22.04

MAINTAINER Tom Looby <loobytp@ornl.gov>

# environment variables
ENV runMode docker
ENV AppDir /root
ENV APPDIR /root
ENV DEBIAN_FRONTEND noninteractive
ENV PYTHONPATH /usr/lib/freecad-python3/lib

#set up paths for libs
ENV LD_LIBARY_PATH $LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu/qt5
ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu/netgen
ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:/root/opt/mafot/lib
#ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:/root/opt/paraview/lib #comment to prevent qt5 conflicts
ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:/root/opt/openfoam/lib
ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:/root/opt/openfoam/platforms/linux64GccDPInt32Opt/lib
ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:/root/opt/openfoam/platforms/linux64GccDPInt32Opt/lib/sys-openmpi
ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:/root/opt/swak4foam/lib
ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:/root/opt/openfoam/platforms/linux64GccDPInt32Opt/lib/dummy
ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:/root/opt/open3d/lib:/root/opt/open3d/lib/python_package

#set up paths for bins
ENV PATH $PATH:/root/opt/mafot/bin
ENV PATH $PATH:/root/opt/paraview/bin
ENV PATH $PATH:/root/opt/openfoam/bin
ENV PATH $PATH:/root/opt/openfoam/platforms/linux64GccDPInt32Opt/bin
ENV PATH $PATH:/root/opt/swak4foam/bin
ENV PATH $PATH:/root/opt/open3d/bin

# install system-wide deps for HEAT
RUN apt-get -yqq update
RUN apt-get -yqq install python3
RUN apt-get -yqq install python3-pip
RUN apt-get -yqq install python3-pkg-resources
RUN apt-get -yqq install python3-distutils
#RUN apt-get -yqq install mdsplus-python
#RUN apt-get -yqq install libfreecad-python3-0.19
RUN apt-get -yqq install software-properties-common
RUN add-apt-repository ppa:freecad-maintainers/freecad-daily
RUN apt-get update
RUN apt-get install -yqq freecad-daily-python3
RUN apt-get -yqq install coreutils
RUN apt-get -yqq install libnglib-6.2
RUN apt-get -yqq install libcurl4
RUN apt-get -yqq install nano
RUN apt-get -yqq install git
RUN apt-get -yqq install gfortran
RUN apt-get -yqq install libunwind-dev
RUN ln -s /usr/lib/x86_64-linux-gnu/libunwind.so /usr/lib/x86_64-linux-gnu/libunwind.so.1

# copy context
WORKDIR /root
COPY . .

# fetch app specific deps
RUN python3 -m pip install --upgrade pip
RUN python3 -m pip install -r requirements.txt

# expose port
EXPOSE 8050

# start app
CMD [ "python3", "./source/HEAT/launchHEAT.py", "--a", "0.0.0.0", "--p", "8050", "--m", "g" ]

Thanks for the assistance.

Prentice

[reidpr]

So the Dockerfile depends on a lot of ORNL context that I don't have.

However, I can't reproduce your errors with the image pulled from Docker Hub. I wonder if it's something in your environment or your dotfiles? Try this:

$ ch-run --unset-env='*' --set-env plasmapotential/heat:v3.0.1 -- ldd /usr/lib/freecad-daily-python3/lib/libFreeCADBase.so | fgrep libQt5Core.so.5
	libQt5Core.so.5 => /lib/x86_64-linux-gnu/libQt5Core.so.5 (0x00007fa80993b000)

[pbisbal1]

Reid,

Thanks so much for your help. I'll play around with --unset-env and --set-env look at my rc files and see what I can find. It may take a couple of days before I can get back to you, though - I have several deadlines staring me in the face right now.

Prentice

[pbisbal1]

Reid,

Thanks for your help. That seems to be the problem. On our cluster, even with a minimal environment, running the command you provided still results in a "not found" for when I run ldd:

$ ch-run --unset-env="*" --set-env plasmapotential/heat:v3.0.1 ldd /usr/lib/freecad-daily-python3/lib/libFreeCADBase.so | fgrep libQt5Core.so.5 
	libQt5Core.so.5 => not found

But that same command provides more promising results on my desktop system:

$ ch-run --unset-env="*" --set-env plasmapotential/heat:v3.0.1 ldd /usr/lib/freecad-daily-python3/lib/libFreeCADBase.so | fgrep libQt5Core.so.5 
	libQt5Core.so.5 => /lib/x86_64-linux-gnu/libQt5Core.so.5 (0x00007fda4cb8e000)

My desktop is a running a free rebuild of RHEL 8 (Springdale 8), which has the qt5-qtbase-5.15 RPM installed, whereas the cluster node where I'm getting the error has qt5-qtbase-5.9.7 installed. Is that the issue? My CentOS 7 host doesn't have a new enough version of Qt5?

Prentice

[pbisbal1]

That makes sense, given my investigations, but I'm new to using containers, and I thought the idea was that a container would contain all the libraries it needed so issues like this wouldn't happen. Is that not the case? If so, someone has been overselling containers! ;)

[reidpr]

Hello Prentice,

The host libraries shouldn't matter, nor the host environment in the case of --unset-env='*' --set-env (note single quotes, I'd have to look up if single/double matters), but I did notice that the Dockerfile does a lot of LD_LIBRARY_PATH stuff, which is not a best practice. (Best practice is to just install into standard paths in the container.)

idea was that a container would contain all the libraries it needed so issues like this wouldn't happen

That is in fact the idea, but it can be defeated. I wonder if that's what happened here.

Do you have site support that could sit with you and debug in detail?

HTH,
Reid

[pbisbal1]

Reid,

Do you have site support that could sit with you and debug in detail?

I am the site support! ;) I'm a Sr HPC Engineer, but this is my first foray into working with containers.

The user I'm working with mentioned something about the linking problem we're seeing being related to how old the kernel is on our cluster (CentOS 7), but I haven't had a chance to independently research and confirm that yet due to competing priorities.

Prentice

[reidpr]

I am the site support! ;)

😂 Why don’t you drop me a line at reidpr@lanl.gov and we’ll see if we can get to the bottom of this.

It’s plausible that the container uses system calls not present in your cluster’s kernel, but my guess is that would manifest as ENOSYS rather than linking problem?