Failing to install pyEMsoft on Linux Ubuntu 20.04

[hakonanes]

Tried installing pyEMsoft from the latest commit 6606859, but failed. Can some get pyEMsoft running on Ubuntu? If so, what does your Python env (Anaconda/venv) look like?

I've followed the installation instructions (https://pyemsoftreadthedocs.readthedocs.io/en/latest/Installation.html).

From an amateur's glance, I believe I've found some minor things in Source/pyEMsoft/run_pyEMsoft.sh.in and kind_map which should be changed:

1. Explanation before code (line 47) should read: "THIS FILE IS AUTOMATICALLY... LOCATED AT @EMsoft_SOURCE_DIR@/Source/pyEMsoft/run_pyEMsoft.sh.in", adding the Source
2. CLsupport.f90 is not located together with the other source files in EMsoftLib, but in its own EMOpenCLLib directory? Therefore it must either be added both places, or the bash script must be updated.
3. Locations of local.f90 and stringconstants.f90 is not EMsoftBuild/Debug/EMsoft/EMsoftLib, but EMsoftBuild/Debug/EMsoftLib. Line 106 in run_pyEMsoft.sh.in should be updated.
4. The kind_map misses the character entries '34':'char' and '38':'char'? I've gotten these lines at the bottom of build_error.log:

RuntimeError: Unknown combination of type "character" and kind "34" - add to kind map and try again
f90wrap: RuntimeError('Unknown combination of type "character" and kind "34" - add to kind map and try again')
         for help use --help

and, after adding 34, I get the same error with 38.

After addressing this and the source file locations, the script runs for ~1-2 hours without any errors being printed to stdout. However, Python complains saying that _pyEMsoft is not found. Looking in build_error.log, I find these lines at the bottom:

analyzevars: charselector={'len': '1'} unhandled.analyzevars: charselector={'len': '1'} unhandled.analyzevars: charselector={'len': '1024'} unhandled.analyzevars: charselector={'len': '1024'} unhandled.analyzevars: charselector={'len': '1024'} unhandled.analyzevars: charselector={'len': '1'} unhandled.analyzevars: charselector={'len': '1'} unhandled.analyzevars: charselector={'len': '1'} unhandled.analyzevars: charselector={'len': '1'} unhandled.analyzevars: charselector={'len': '1024'} unhandled.analyzevars: charselector={'len': '1024'} unhandled.analyzevars: charselector={'len': '1024'} unhandled.analyzevars: charselector={'len': '1'} unhandled.analyzevars: charselector={'len': '1'} unhandled.analyzevars: charselector={'len': '6'} unhandled.append_needs: unknown need 'char'
error: Command "gcc -pthread -B /home/hakon/miniconda3/envs/kp-dev/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DSCIPY_MKL_H -DHAVE_CBLAS -I/home/hakon/kode/emsoft/EMsoftBuild/Debug/EMsoftLib -I/home/hakon/kode/emsoft/EMsoftBuild/Debug/EMsoft/EMsoftHDFLib -I/home/hakon/kode/emsoft/EMsoft_SDK/hdf5-1.10.5-Debug/include/static -I/home/hakon/kode/emsoft/EMsoft_SDK/CLFortran-0.0.1-Debug/include -I/home/hakon/kode/emsoft/EMsoft_SDK/jsonfortran-4.2.1-Debug/include -I/home/hakon/kode/emsoft/EMsoft_SDK/fftw-3.3.8/include -I/home/hakon/miniconda3/envs/kp-dev/include -I/tmp/tmpic0h8u7u/src.linux-x86_64-3.8 -I/home/hakon/miniconda3/envs/kp-dev/lib/python3.8/site-packages/numpy/core/include -I/home/hakon/miniconda3/envs/kp-dev/include/python3.8 -c /tmp/tmpic0h8u7u/src.linux-x86_64-3.8/_pyEMsoftmodule.c -o /tmp/tmpic0h8u7u/tmp/tmpic0h8u7u/src.linux-x86_64-3.8/_pyEMsoftmodule.o -MMD -MF /tmp/tmpic0h8u7u/tmp/tmpic0h8u7u/src.linux-x86_64-3.8/_pyEMsoftmodule.o.d" failed with exit status 1

I guess they are related to what's explained in the installation guide: https://pyemsoftreadthedocs.readthedocs.io/en/latest/Installation.html#debugging, but I have too little knowledge to tell.

[ChaoyiZhu93]

thank you. sorry for the late follow-up here. this should be resolved

a33f0b4

[hakonanes]

Hi @ChaoyiZhu93,

I'm sorry, even with the latest commits (source code version: 5_0_20210320_) I still can't install pyEMsoft on my Ubuntu 20.04:

> ./run_pyEMsoft.sh 
 run_pyEMsoft.sh: copying source files into place
 run_pyEMsoft.sh: executing f90wrap
 run_pyEMsoft.sh: executing f2py-f90wrap ... this can take a very long time (several hours) ...
 run_pyEMsoft.sh: cleaning up
 run_pyEMsoft.sh: pyEMsoft wrapper .so file build completed
 copying examples and unittests folders into build folder
mv: klarte ikke å hente informasjon om '*.so': Fila eller mappa finnes ikke
 Done.

"klarte ikke å hente informasjon om '*.so': Fila eller mappa finnes ikke" is norwegian for "couldn't get information on '*.so': File or directory doesn't exist"

> python setup.py install
 running install
 [...]
 Finished processing dependencies for EMsoft==1.0
> python -c "from EMsoft import pyEMsoft"
 Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/hakon/kode/emsoft/EMsoftBuild/Debug/pyEMsoft/EMsoft/pyEMsoft.py", line 2, in <module>
    import _pyEMsoft
 ModuleNotFoundError: No module named '_pyEMsoft'

I guess there is something wrong with the linked libraries? In the installation guide

To check what dynamic libraries are linked to the shared library file (.so), you can use ldd *.so to check the dynamically linked dynamic libraries.

where should I execute "ldd *.so"?

[ChaoyiZhu93]

thank for testing this out. do you have the error message output file when you compile ./run_pyEMsoft.sh ? this will give us some insight to the .so problem.

i am wondering if this is an issue with the name of lib folder in fftw-3.3.8 being lib64 whereas our .sh script forgot to take that into account.

[hakonanes]

The last two lines in EMsoftBuild/Debug/pyEMsoft/logs/build_error.log (build_error.log), which is the file I assume you want, are

analyzevars: charselector={'len': '1'} unhandled.analyzevars: charselector={'len': '1'} unhandled.analyzevars: charselector={'len': '1024'} unhandled.analyzevars: charselector={'len': '1024'} unhandled.analyzevars: charselector={'len': '1'} unhandled.analyzevars: charselector={'len': '1'} unhandled.analyzevars: charselector={'len': '1'} unhandled.analyzevars: charselector={'len': '1'} unhandled.analyzevars: charselector={'len': '1024'} unhandled.analyzevars: charselector={'len': '1024'} unhandled.analyzevars: charselector={'len': '1024'} unhandled.analyzevars: charselector={'len': '1'} unhandled.analyzevars: charselector={'len': '1024'} unhandled.analyzevars: charselector={'len': '1024'} unhandled.analyzevars: charselector={'len': '1024'} unhandled.analyzevars: charselector={'len': '1024'} unhandled.analyzevars: charselector={'len': '1'} unhandled.analyzevars: charselector={'len': '6'} unhandled.append_needs: unknown need 'char'
error: Command "gcc -pthread -B /home/hakon/miniconda3/envs/pyem/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DNO_ATLAS_INFO=1 -DHAVE_CBLAS -DNPY_DISABLE_OPTIMIZATION=1 -I/home/hakon/kode/emsoft/EMsoftBuild/Debug/EMsoftLib -I/home/hakon/kode/emsoft/EMsoftBuild/Debug/EMsoftHDFLib -I/home/hakon/kode/emsoft/EMsoftBuild/Debug/EMOpenCLLib -I/home/hakon/kode/emsoft/EMsoft_SDK/hdf5-1.10.7-Debug/include/static -I/home/hakon/kode/emsoft/EMsoft_SDK/CLFortran-0.0.1-Debug/include -I/home/hakon/kode/emsoft/EMsoft_SDK/jsonfortran-4.3.0-Debug/include -I/home/hakon/kode/emsoft/EMsoft_SDK/fftw-3.3.8/include -I/home/hakon/miniconda3/envs/pyem/include -I/tmp/tmpnblptfz6/src.linux-x86_64-3.8 -I/home/hakon/miniconda3/envs/pyem/lib/python3.8/site-packages/numpy/core/include -I/home/hakon/miniconda3/envs/pyem/include/python3.8 -c /tmp/tmpnblptfz6/src.linux-x86_64-3.8/_pyEMsoftmodule.c -o /tmp/tmpnblptfz6/tmp/tmpnblptfz6/src.linux-x86_64-3.8/_pyEMsoftmodule.o -MMD -MF /tmp/tmpnblptfz6/tmp/tmpnblptfz6/src.linux-x86_64-3.8/_pyEMsoftmodule.o.d" failed with exit status 1

i am wondering if this is an issue with the name of lib folder in fftw-3.3.8 being lib64 whereas our .sh script forgot to take that into account.

My EMsoft_SDK/fftw-3.3.8 contains a lib directory with the .so shared library links, not a lib64 directory, so I think that's not it?

[ChaoyiZhu93]

thanks for your reply. i think the problem is that currently it only works for the Release version of EMsoft (built with BUILD_SHARED_LIBS=ON). i am running some tests now and i should be able to update it shortly.

[ChaoyiZhu93]

the updated installation instruction is here: pyEMsoft. it is important to build the shared libraries when you compile the EMsoft.

[hakonanes]

i think the problem is that currently it only works for the Release version of EMsoft

As in CMake build type "Release", -DCMAKE_BUILD_TYPE=Release? If not, what is considered the release version for Linux?

the updated installation instruction is here: pyEMsoft.

Thanks, I got the error above after following those instructions, including building of shared libraries during compilation, which were updated late December last year.

But could you tell me in which EMsoft directory I should execute ldd *.so to produce the shared libraries list from the installation instructions?

[ChaoyiZhu93]

could you try to pull the recent commits #142 and give this a try? i should also work for the Debug mode now? you can only check with the ldd comment if the .so file is compiled and should be located in the pyEMsoft/EMsoft folder.

[hakonanes]

Thanks a lot for looking into this, trying it now.

Which directory do you mean with "pyEMsoft/EMsoft"? I have three directories named pyEMsoft in with EMsoftBuild/Debug/pyEMsoft, EMsoft/Source/pyEMsoft and EMsoft/Source/EMsoftWrapperLib/pyEMsoft. And the command I should use in that directory is "ldd *.so"?

[hakonanes]

I'm sorry, I misunderstood, I thought you meant I should check for linked libraries within the pyEMsoft directory, but I guess what you mean is that they should be located in that directory. I'm still uncertain from which directory I should check for linked libraries. I'm sorry, I'm not familiar with linked/shared libraries at all.

[ChaoyiZhu93]

No problem. we are still testing out some of the features of this so thank you for testing this out. Assuming that you have the pyEMsoft project compiled successfully, that .so file should be located in the Build folder e.g. \home\chaoyi\EMsoftBuild\Release\pyEMsoft\EMsoft.

Sometimes, we found that libraries were not linked correctly so what I did was to go to that folder and do: ldd _pyEMsoft.cpython-38-x86_64-linux-gnu.so. this can help you determine which dynamically linked libraries are not linked.

For instance, if the libEMsoftLib.so is not linked, you can do "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/chaoyi/EMsoftBuild/Release/Bin" to point to the Bin folder containing the file.