The S-function does not compile with Matlab R2018b

[diegoferigo]

When compiling the Mex target, the compilation fails with the following error:

Invalid MEX-file '/.../BlockFactory.mexa64': Gateway function is missing.

This happens only on Matlab R2018b. The same commit builds fine on (at least) R2018a.

[diegoferigo]

From what it seems, the entrypoint function of the mex is missing. More information at mexFunction.

TL;DR
Matlab looks for this symbol when it tries to load a mex library. If it is present, it calls it and proceeds with the computation.

There are already few bug reports I found, specifically:

• Matlab R2018b DIPlib/diplib#13
• https://gitlab.kitware.com/cmake/cmake/issues/18391 (Fix)
• Error after using compiled mex files (CMake / VS 2015): Invalid MEX-file ... Function mexFunction is missing. alecjacobson/gptoolbox#70

[traversaro]

As the mexFunction entry point is used also in SWIG bindings, I guess this problem affects also iDynTree and YARP matlab bindings. How do you suggest to proceed? Vendoring the latest FindMatlab.cmake will work, but we can also manually add the line:

set_target_properties(${${prefix}_NAME}
                      PROPERTIES
                      LINK_FLAGS "${_previous_link_flags} /EXPORT:mexFunction")

outside the matlab_add_mex function.

[traversaro]

This happens only on Windows or also on Linux? It applies also to 2019a ?

[traversaro]

Ok, 2019a is not out yet. : )

[diegoferigo]

Yesterday on @aikolina setup (Ubuntu 16.04) we added:

set_target_properties(Mex PROPERTIES COMPILE_FLAGS "-fvisibility=default")

For sure this works, I am not yet sure if this is too drastic (though it's not a big issue since we've never hidden any symbol in this project). I am waiting @aikolina pc to investigate deeper.

[diegoferigo]

Further infos on the reason why the symbols are usually hidden:

Symbol clash in a MEX target

By default, every symbols inside a MEX file defined with the command matlab_add_mex() have hidden visibility, except for the entry point. This is the default behaviour of the MEX compiler, which lowers the risk of symbol collision between the libraries shipped with Matlab, and the libraries to which the MEX file is linking to. This is also the default on Windows platforms.

However, this is not sufficient in certain case, where for instance your MEX file is linking against libraries that are already loaded by Matlab, even if those libraries have different SONAMES. A possible solution is to hide the symbols of the libraries to which the MEX target is linking to. This can be achieved in GNU GCC compilers with the linker option -Wl,--exclude-libs,ALL.

[traversaro]

Does this mean that the upstream FindMatlab.cmake is broken for Matlab 2018b + Ubuntu 16.04/18.04 ?

[traversaro]

If there was an equivalent to MSVC's /EXPORT function in GCC/Clang/ld that would be the ideal solution, these lines in the old FindMatlab in iDynTree see to implement something similar:
https://github.com/robotology/idyntree/blob/4a231725512978216e316d314ff4fe5d82aee5ea/cmake/FindMatlab.cmake#L927 .

However, it seems that the visibility of the mexFunction symbols is handled by the DLL_EXPORT_SYM macro: what happened in Matlab 2018b? The DLL_EXPORT_SYM is not used anymore in the headers that define the function mexFunction?

[traversaro]

If you have time, can you check inside the mex.h and related headers how the mexFunction declaration changed between 2018a (or earlier) and 2018b ?

[diegoferigo]

Does this mean that the upstream FindMatlab.cmake is broken for Matlab 2018b + Ubuntu 16.04/18.04?

No my last quote is the reason why CMake hides by default all symbols but mexFunction. The fact that also mexFunction is hidden seems a problem of Matlab headers. Here below the relevant sections:

// R2018a
#ifdef _WIN32
#       define DLL_EXPORT_SYM __declspec(dllexport)
#       define SUPPORTS_PRAGMA_ONCE
#elif __GNUC__ >= 4
#       define DLL_EXPORT_SYM __attribute__ ((visibility("default")))
#       define SUPPORTS_PRAGMA_ONCE
#else
#       define DLL_EXPORT_SYM
#endif

#ifdef DLL_EXPORT_SYM
# define MEXFUNCTION_LINKAGE EXTERN_C DLL_EXPORT_SYM
#else
# ifdef MW_NEEDS_VERSION_H
#  include "version.h"
#  define MEXFUNCTION_LINKAGE EXTERN_C DLL_EXPORT_SYM
# else
#  define MEXFUNCTION_LINKAGE EXTERN_C
# endif
#endif

// R2018b
#ifdef _MSC_VER
# define MWMEX_EXPORT_SYM __declspec(dllexport)
#elif __GNUC__ >= 4
# define MWMEX_EXPORT_SYM __attribute__ ((visibility("default")))
#else
# define MWMEX_EXPORT_SYM
#endif

#ifdef MW_NEEDS_VERSION_H
# define MEXFUNCTION_LINKAGE LIBMWMEX_API_EXTERN_C MWMEX_EXPORT_SYM
#else
# define MEXFUNCTION_LINKAGE LIBMWMEX_API_EXTERN_C
#endif

[diegoferigo]

Refer to the upstream CMake fix, particularly:

if(NOT ${Matlab_VERSION_STRING} VERSION_LESS "9.5") # For 9.5 (R2018b) (and newer?)
      target_compile_options(${${prefix}_NAME} PRIVATE "-fvisibility=default")
      # This one is weird, it might be a bug in <mex.h> for R2018b. When compiling with
      # -fvisibility=hidden, the symbol `mexFunction` cannot be exported. Reading the
      # source code for <mex.h>, it seems that the preprocessor macro `MW_NEEDS_VERSION_H`
      # needs to be defined for `__attribute__ ((visibility("default")))` to be added
      # in front of the declaration of `mexFunction`. In previous versions of MATLAB this
      # was not the case, there `DLL_EXPORT_SYM` needed to be defined.
      # Adding `-fvisibility=hidden` to the `mex` command causes the build to fail.
      # TODO: Check that this is still necessary in R2019a when it comes out.
endif()

[traversaro]

Why we don't simply define MW_NEEDS_VERSION_H ?

[traversaro]

It would be cool to see which compiler commands are actually called by the mex command when compiling (see https://it.mathworks.com/help/matlab/ref/mex.html), for example by launching matlab under strace.

[diegoferigo]

Reading comments from the issues I linked, it does not seem the right way to fix this. If CMake went to this direction I would apply the same fix in order to match the behavior of recent CMake version (I guess, >= 3.13.0, not sure if it will be backported).

[traversaro]

Reading comments from the issues I linked, it does not seem the right way to fix this.

Why?

If CMake went to this direction I would apply the same fix in order to match the behavior of recent CMake version (I guess, >= 3.13.0, not sure if it will be backported).

Ok, this make sense even if I suspect the fix is sub-optimal.

[diegoferigo]

Why?

I am not sure if it is used in other places nor what can happen now or in the future by enabling it. I think that the solution of exposing more symbols knowing for sure that there is no symbols collision (knowing our code) is safer than enabling an option that can insert code that we do not control (and are not aware since we cannot test many Matlab versions).

[traversaro]

. I think that the solution of exposing more symbols knowing for sure that there is no symbols collision (knowing our code) is safer than enabling an option that can insert code that we do not control (and are not aware since we cannot test many Matlab versions).

I see. I think the definitive fix is to check which kind of preprocessor defines the mex command is passing to the compiler (either using strace or passing the -v option as defined in https://it.mathworks.com/matlabcentral/answers/84440-how-can-i-see-the-command-line-options-that-the-mex-command-passes-to-the-c-compiler): the mex command by definition (and because it is actually tested by Mathworks) is passing the intended and working flags, and in CMake we should simply try to replicate that. I did not see the full Matlab 2018b mex.h, but I suspect that the mex command is actually defining MW_NEEDS_VERSION_H (or something that results in MW_NEEDS_VERSION_H being defined).

[diegoferigo]

It looks that mex is a matlab command. In order to use it, we should figure out how to compile our S-Function from matlab command line. If this verbose flag as I think prints the complete set of options at the very beginning (as VERBOSE=1 of make), we don't care to specify the right includes and linked libraries.

The only user that as of today has the R2018b version of Matlab is @aikolina as far as I know.

[traversaro]

I think we just need to run mex on any mex library (for example a one of the matlab examples, see the examples in https://it.mathworks.com/help/matlab/ref/mex.html), and check which flags are passed: we do not need to compile one of our specific projects. From my understanding, this problem is shared by all the projects that compile mex libraries (such as our YARP and iDynTree SWIG-based bindings) and so I am concerned in finding a solution that works fine for all our use cases (and I suspect at that point also CMake fix will need to be changed, if it turns out that mex defines something that results in MW_NEEDS_VERSION_H being defined).

[traversaro]

The only user that as of today has the R2018b version of Matlab is @aikolina as far as I know.

If @aikolina can do run the mex example in https://it.mathworks.com/help/matlab/ref/mex.html with the -V flag and could copy paste the output here, it would be great! Furthermore, could you check if the iDynTree/YARP matlab bindings are working correctly?

[traversaro]

Anyhow, I just realized that I had run an old copy of https://github.com/robotology/mex-wholebodymodel on a Matlab 2018b recently, and everything was working fine. Note that mex-wholebodymodel and also iDynTree YARP bindings actually use a vendored version of FindMatlab.cmake, that is probably different from the one that blockfactory is using on @aikolina machine.

[diegoferigo]

@traversaro Probably the vendored FindMatlab.cmake does not hide by default the symbols, and everything runs fine for this reason.

As soon as we will be back we should do the check on the definitions you were mentioning.

[traversaro]

@traversaro Probably the vendored FindMatlab.cmake does not hide by default the symbols, and everything runs fine for this reason.

Apparently this is not the case: https://github.com/robotology/mex-wholebodymodel/blob/master/cmake/FindMatlab.cmake#L909 , but it is possible that that code is not working as intended.

[diegoferigo]

Here is the map file provided in Matlab 2018a:

# This is the symbol export map file for glibc Linux.  The file specifies
# that only the mexFunction symbol should have global scope,
# and that all others should be local.
# Copyright 2004-2017 The MathWorks, Inc.
MEX {
        global:
                mexFunction;
                mexCreateMexFunction;
                mexDestroyMexFunction;
                mexFunctionAdapter;
                mexfilerequiredapiversion;
        local:
                *;
};

At least this is provided directly from Mathworks. From the MATLAB Central thread it seems that this is not the case for Windows.

[traversaro]

Indeed, we should check also what they are doing on macOS, as that is not reported in the thread.

From the MATLAB Central thread it seems that this is not the case for Windows.

The equivalent functionality in Windows is provided by the .def file, but the command line /EXPORT option should provide the same functionality in a more compact way.

[traversaro]

It is interesting that for some reason they document to export mexFunction and mexfilerequiredapiversion in both Linux and Windows, but mexCreateMexFunction, mexDestroyMexFunction and mexDestroyMexFunction only on Linux. I wonder if that is just something that was accidentally omitted in the MATLAB Answers docs.

[diegoferigo]

Next week I plan to release the first release. If you don't have anything against it, I would temporary go for the visibility fix that is quick and effective, and leave more in-depth investigations and tests to the release after. @traversaro

[traversaro]

Ok for me.

[diegoferigo]

PR #36 should fix this issue but let's keep it open to check if the situation can be improved further.

[lrapetti]

I had a similar issue when compiling BlockFactory in MacOS 10.13.6 with MATLAB_R2018a.

The error I was getting is the following:

[ 22%] Performing build step for 'BlockFactory'
[  9%] Built target mxpp
[ 28%] Built target shlibpp
[ 71%] Built target Core
[ 80%] Built target SimulinkCoder
[ 85%] Linking CXX shared library ../../lib/BlockFactory.mexmaci64
Undefined symbols for architecture x86_64:
  "_mexfilerequiredapiversion", referenced from:
     -exported_symbol[s_list] command line option
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

@diegoferigo helped me to debug the problem, and it is currently fixed by commenting out _mexfilerequiredapiversion from MATLAB_R2018a.app/extern/lib/mac/c_exportsmexfileversion.map

[diegoferigo]

Here some more details. @lrapetti configuration arrives at this line of the FindMatlab.cmake file:

blockfactory/cmake/FindMatlab.cmake

Line 1048 in a05e6d0

set(_ver_map_files ${Matlab_EXTERN_LIBRARY_DIR}/c_exportsmexfileversion.map)

since his matlab is recognized as R2018a 9.3. The difference between mexFunction.map and c_exportsmexfileversion.map is only the problematic symbol. It might be a upstream CMake bug, even if it is strange because that symbol should be exported starting from R2016b accordingly to the comments in FindMatlab.cmake, quite an old release.

[nunoguedelha]

@diegoferigo @traversaro @lrapetti , I just got the same problem on MacOS Sierra with Matlab 2017b. commenting the mentioned line also worked. Any observed side effects?

[diegoferigo]

Any observed side effects?

Not that I know. Since blockfactory has a vendored version of FindMatlab.cmake, we can fix it locally by editing the following section:

blockfactory/cmake/FindMatlab.cmake

Lines 1045 to 1049 in a05e6d0

	if(${Matlab_VERSION_STRING} VERSION_LESS "9.1") # For versions prior to 9.1 (R2016b)
	set(_ver_map_files ${Matlab_EXTERN_LIBRARY_DIR}/mexFunction.map)
	else() # For 9.1 (R2016b) and newer
	set(_ver_map_files ${Matlab_EXTERN_LIBRARY_DIR}/c_exportsmexfileversion.map)
	endif()

and changing the check on the Matlab version. However, it will be just a workaround since this problem might appear also on other Matlab versions and we cannot test more than the ones that we own. What do you think @traversaro?

[traversaro]

I would fix the blockfactory vendored FindMatlab.cmake's file, and open an issue in CMake reporting the problem.

[traversaro]

If I understood correctly, @Giulero has Ubuntu 18.04 and Matlab 2019a and he is able to run the WB-Toolbox without problems. Probably the workaround in #4 (comment) is necessary just for macOS (or at least it is not necessary on Linux)?

[traversaro]

MathWorks never replied to my comment in #4 (comment) . Given how this answer system works, it may be worth to try to ask the same question in a new question, instead of just commenting an old answered question.

[diegoferigo]

Maybe they fixed it in 2019a? I still didn't update, they might have changed piece of code reported above in #4 (comment).

[traversaro]

@Giulero (or anyone that has 2019a) can you check what the mex.h file contains in the portion mentioned in #4 (comment) ?