AppImage build failing

[terminator356]

It's a pesky dependency thing. It's super 'touchy'. Today it said one ui file then
after simply altering one line of actual code it said another ui file.
Currently, you can see it's trying to compile the mixer, which depends on the routing dialog
found in the components folder, before the components are built.
I hesitate to call it a 'circular' dependency because:

1. I fixed a lot of that stuff several months ago, but not all, but I got most of the 'meat' of it,
   by doing a whole lotta forward referencing in all the header files.
2. So far, I can find no circular dependency that would cause the mixer to want to be built
   before the components.
3. I have just built in both Release and Debug mode, with -j3, and no errors -
   in fact, strange, here it builds the mixer before the components:
   Scanning dependencies of target mixer
[ 55%] Building CXX object muse/mixer/CMakeFiles/mixer.dir/amixer.o
[ 55%] Generating moc_splitter.cpp
[ 55%] Generating moc_synctoolbar.cpp
[ 55%] Generating moc_tb1.cpp
[ 55%] Generating moc_tools.cpp
[ 55%] Generating moc_trackinfo_layout.cpp
[ 55%] Generating moc_tracks_duplicate.cpp
[ 55%] Generating moc_ttoolbutton.cpp
[ 55%] Generating moc_unusedwavefiles.cpp
[ 56%] Generating moc_view.cpp
[ 56%] Generating moc_visibletracks.cpp
[ 56%] Generating moc_vst_native_editor.cpp
Scanning dependencies of target components
[ 56%] Building CXX object muse/components/CMakeFiles/components.dir/moc_aboutbox_impl.o
[ 56%] Building CXX object muse/components/CMakeFiles/components.dir/moc_appearance.o
[ 56%] Building CXX object muse/mplugins/CMakeFiles/mplugins.dir/mitplugin.o
[ 56%] Building CXX object muse/components/CMakeFiles/components.dir/moc_arrangercolumns.o
[ 56%] Building CXX object muse/mixer/CMakeFiles/mixer.dir/astrip.o
   etc...

I've been through this many times, except with actual circular dependencies,
many of which I fixed.
Recall a user's issue (maybe still there) where only -j3 triggered it, but I couldn't reproduce it.

For these reasons, since it seems so 'touchy', and until we can figure out what's going on,
it seems like a fickle race condition so I recommend trying -j1 or -j4 in the AppImage
script instead of -j3. I didn't want to mess with it in case I break something.

Please give it a try, I'll bet that they work.

[terminator356]

So far, I can find no circular dependency that would cause the mixer to want to be built
before the components.

Er, maybe that's not exactly true. The app depends on the mixer, so maybe that's why it's
building first? But why is the AppImage build getting it wrong when it is clearly the
same order as observed here - mixer then components? Possibly multi-job related, one job
is in a race with the others, jobs are interrupting another job in progress, on different hardware?

You know, for the longest time, these errors have always been with those darn ui files.
I wonder if we're getting the order wrong in the cmake files, like maybe QT5_WRAP_UI the .ui files
before QT5_WRAP_CPP the .h files or something?

[kybos]

I'm afraid I'm no expert in this area...
Will try with different number of threads.
On my own machine I build with 32 threads and never had any issues, funny.

[kybos]

With -j4 it failed in another place.
With -j1 it worked.
But I guess this is no real solution, nobody is building with 1 thread only nowadays...

[kybos]

-j2 is failing too.

[terminator356]

Thanks!

nobody is building with 1 thread only nowadays

Exactly. Except for the AppImage builder ! So... we uh could get away with that here ... he he

Glaring omission: Sorry I forgot to mention: Most of the time these problems are
caused by ui files and/or related code files being in different places, scattered all over the place.
I fixed a lot of it by moving several ui/code modules together into the components folder.
Because of the complex dependencies, the build wants to see most ui/code files together in the
components folder.
Here is a case where that is happening. The routing dialog is in the components folder
but the mixer code is in a different folder.
Moving the routing dialog into the mixer folder would likely make things worse
since other things depend on that routing dialog as well.
Therefore one solution as mentioned would be to move the mixer code files into the components folder.
After all, the mixer really is just a component.
I believe however that to really make things right, to prevent this from ever happening,
most of the other folders would also need to be moved into the components folder as well.
In other words, one giant source folder !
I hate to have to do that. I hope we can find a way to allow the ui/code files to exist
in different folders like we have now.
But it seems really tricky with all the dependencies (circular or not).
This wants to build before that, that wants to build before this etc.

I'll check more later tonight. Off to work, that four letter word...

[kybos]

The AppImage build is not the problem, it doesn't matter if it takes a little longer with only one thread. But it runs in a clean environment so when it fails, it's quite probable that users will have similar problems too.

I tried a couple of things, no success (except for my better understanding of cmake perhaps...).
I changed the build to -j3 again, in case you want to test something with this setting.

[terminator356]

Hang in there, trying some things.
The use of cmake subdirs is deprecated. Replaced with add_subdirectory.
We use subdirs in a few places including the top level.
The one in muse/CMakeLists.txt is of interest.

Read this interesting cmake help note for add_subdirectory which is not found for subdirs:

The CMakeLists.txt file in the specified source directory will be processed immediately
by CMake before processing in the current input file continues beyond this command.

It seems to be having an effect. It seems to demand the correct order - which is good !
But I'm still worried. All the moc and ui processing still seems to have a mind of its own.
It seems to go off in its own job (with multi-job) haphazardly interspersed with all the other source code,
and doesn't seem to care that something down the line needs portions of it to be ready first.
Sure doesn't seem like any kind of 'automatic dependency ordering' is working well there.
For the source code yes, but those mocs and uis, hm seems odd...

Till tomorrow...

[kybos]

I replaced the subdirs command by add_subdirectory in all relevant cmake files. The syntax and the behaviour are slightly different, but I think it's not related to the sequence of execution. The difference is better described here:

ADD_SUBDIRECTORY() processes the subdir at the time it is called, whereas SUBDIRS() pushes the dirs onto a list which is processed at the end of the current CMakeLists file - this is the old behaviour and some vars are initialized 'out of order', or at least in unexpected order.

The build problems obviously are caused by the fact that the specific qt files/includes (ui, moc...) do not exist at design time. Being generated dynamically at compile time, cmake is not able to calculate the related dependencies correctly. I added explicit dependency to the mixer cmake file - add_dependency(mixer components). Now the build works again normally, AFAICS (with -j3 too).

[terminator356]

Oh very nice! Good catch. Good work!
I wondered that as well - how to force a dependency.

We may need that add_dependency for some of the other folders, for example the midi editor folder.

Will try your fixes tonight...

[kybos]

I added the dependencies to midiedit and also waveedit (there were 2 open issues about that one).
The command is e.g. "add_dependencies(midiedit components)". I had a typo in the command name above.