README-DEV

# @@ TMP "Inconsistent compiler behavior" build2 errors when CI'ing clang 18
# on Windows.
# E.g. https://ci.stage.build2.org/@f58b86b2-9784-4565-afb0-7cc34001c94e/libQt6Widgets/6.7.2/log/x86_64-microsoft-win32-msvc14.3/windows_10-clang_18_llvm_msvc_17.10/default/stage/0.18.0-a.0.20240628122722.68eac2eacdac

# @@ TODO Windows CI logs do now show compilation of libQt6Core sources. Are
#    the logs truncated? Need this to confirm macros are correct on Windows.

Qt release information: https://wiki.qt.io/QtReleasing

Also see build2-packaging/Qt5/README-DEV.

1. The upstream build

Note: See Appendix A for instructions for running the upstream build on
      various operating systems.

Qt5's qmake build has been completely converted to cmake in Qt6.

Scripts in upstream/qtbase/util/cmake/ were used to convert much of the Qt5
build into CMake files. For example, configurejson2cmake.py was used to
convert the Qt5 configure.json files to configure.cmake for each module, and
the Qt5 qmake project files (.pro, .pri) were converted to CMakeLists.txt
using the pro2cmake.py script.

1.1 Upstream build documentation

https://wiki.qt.io/Qt6_Build_System (not much there at the time of writing,
however).

1.2 Macro definitions

Module-specific macros are defined in each module's CMakeLists.txt. Common
ones are defined in various .cmake files in upstream/qtbase/cmake, using the
following variables and functions:

  QT_DEFAULT_PLATFORM_DEFINITIONS
  QT_PLATFORM_DEFINITIONS
  target_compile_definitions
  QT_EXTRA_INTERNAL_TARGET_DEFINES
  defines_for_extend_target
  qt_internal_add_global_definition

This list should be relatively stable but probably better be updated
periodically. Searching upstream/qtbase/cmake/ for some of the macros we
define in libQt6Core/build/common.build should help identify them.

1.3 Configuration system

A module's features are defined in qtbase/src/<module>/configure.cmake as
follows:

  qt_feature("<feature-name>" <visibility>...)

This results in definitions like the following in the module config header
(public or private, depending on the visibility; no visibility means it's an
internal Qt feature only visible to the upstream build):

  #define QT_FEATURE_foo  1 // Enabled feature
  #define QT_FEATURE_bar -1 // Disabled feature

Upstream code checks for features using the QT_CONFIG() macro defined in
qtconfigmacros.h. Every feature "foo" thus tested must have a corresponding
`#define QT_FEATURE_foo {1,-1}` in one of the module
config headers, otherwise there will be a compilation error.

Some features have associated macros that are defined only if the features is
either enabled or disabled. These are always defined in the public module
config header. Then there are "extra" macros which are standalone and can be
defined in the public or private module config header.

These macros are defined using the following functions:

qt_feature_definition(<feature-name> <macro-name>): Define a
                                                    feature-associated macro
                                                    in the public module
                                                    config header
                                                    conditionally based on the
                                                    enabled value of the
                                                    feature.

qt_extra_definition(<macro-name> PUBLIC|PRIVATE): Define a macro in the public
                                                  or private module config
                                                  header.

1.3.1 Downstream module configuration headers

Because we manage these headers manually we have automated the process of
checking that we haven't missed any features or macros defined upstream. This
is necessary because there are currently more than 400 of them and Qt6 is
still being actively developed so features and macros will be added, removed,
and moved (for example, from the public to the private header).

The `check-config-header` script prints all features and macros not handled
both upstream and downstream. Therefore, in order for it to produce usable
output, it is necessary to put every single feature and macro defined upstream
in our downstream headers. Note however that commented-out features and macros
are treated identically to uncommented ones so not every feature and macro
needs to actually be defined.

Many features defined upstream are not actually referenced by any code. These
are typically only used internally by the upstream build. We prefer to leave
these features/macros commented out because this makes it clear that the
feature/macro is unused and obviates the need to have to figure out under what
conditions it should be enabled and writing the additional code that that
would require.

Recommended ordering of features and macros within a downstream module config
header:

- Application/user-level features
  For example: scrollbars.

- Internal Qt library-level features
  For example: threads, concurrent, appstore_compliant.

- Dependencies
  For example: zstd, glib, icu.

- Build-related
  For example: shared, static, debug_and_release, cross_compile, framework

- C and C++ language features
  For example: c99, cxx17, cxx11_filesystem

- Low-level implementation and platform-specific details
  For example: largefile support, signaling_nan, SSE, AVX, AES.

Divergences from upstream

Although QT_CONFIG() is not available in qconfig.h and qtcore-config.h by
default (because qglobal.h includes them before the header that defines
QT_CONFIG(), qtconfigmacros.h), we put our own definition in qconfig.h so that
we can consistently use QT_CONFIG across all module config headers. Although
upstream clearly didn't intend for this macro to be used in the module config
headers, its downstream use is necessary because testing QT_FEATURE_foo macros
directly would just be too verbose and error prone.

Private features that have an associated macro are moved to the public
header. Otherwise the feature macro and its associated macro would be in
different headers which would make manual maintenance impractical. The
associated macro is only defined if the feature is either enabled or disabled
and we'd otherwise not be able express this relationship programmatically
because private features are not available in the public module config
headers. We'd also have to document the relationship between the feature and
its associated macro using comments and add cross-references to the public and
private headers. Furthermore, it seems that if you need a public macro
QT_NO_FOO which is defined if a private feature QT_FEATURE_foo is disabled
then the feature is not really very private anymore as one half of its
potential values are already exposed. I don't think making such features
public downstream is taking it much further. When the feature and associated
macro are in the same header everything is completely clear:

  #define QT_FEATURE_jpeg 1 // Moved from private
  #if !QT_CONFIG(jpeg)
  #  define QT_NO_IMAGEFORMAT_JPEG 1
  #endif

The "Moved from private" comment tells the `check-config-header` script to not
report the feature as being in the downstream public header (otherwise it
would look like a public feature that was removed from upstream). To prevent
`check-config-header` from reporting it as a missing private feature, also
keep it in the private header, but commented out, with a "Moved to public"
comment for the sake of human readers:

  // #define QT_FEATURE_jpeg 1 // Moved to public

In the upstream module feature configuration files (for example,
upstream/qtbase/src/gui/configure.cmake), the `PUBLIC PRIVATE` visibility spec
causes a feature to be defined in both the public and private module headers
(presumably to cause public features to be automatically pulled in when the
private features are included).

However, we treat `PUBLIC PRIVATE` and `PRIVATE PUBLIC` merely as public (that
is, we do not duplicate these features into both the public and private module
config headers) for the following reasons:

a) It's not clear that the `PUBLIC PRIVATE` scheme is even necessary. I asked
   one of the developers who works on the cmake build on IRC what its purpose
   is (in light of the points below) and even he couldn't tell me.

   As far back as Qt5 the upstream intention has clearly been that the module
   config headers should be included only via the corresponding module global
   header:

   Firstly, there are only a handful of places outside of the module global
   headers where a module config header is included directly, but hundreds
   where a global module header is included.

   Secondly, every module's private global header includes its private module
   config header and its public global header, which includes the public
   module config header. (For example, qtwidgetsglobal_p.h includes
   qtwidgets-config_p.h, and qtwidgetsglobal.h, which includes
   qtwidgets-config.h.) Thus the public features are automatically pulled in
   when the private features are included. Including the private module config
   headers directly would defeat this scheme.

   So the automatic pulling in of public features when private features are
   included was already covered by the global module header scheme for a long
   time.

b) Because we manage the module config headers manually it would be very
   tedious to keep every feature duplicated in both files and it would also
   make reading those files much more difficult.

For these reasons we diverge from upstream by including the public module
config header from the private one instead of duplicating `PUBLIC PRIVATE`
features in both headers.

In module config headers we also include those of the lower-level modules
because there are a handful of cases where module config headers are included
directly instead of via the module global headers as is done in 95% of
cases. This does not cause any immediate problems upstream but it causes build
failures for us because we define some features conditionally upon lower-level
modules' features.

1.4 How to identify sources to be moc'd

grep -REl 'Q_OBJ|Q_GADGET|Q_NAMESPACE' upstream/qtbase/src/<module>

This will include headers and source files (C++ and Objective-C++). The
transformations are as follows:

  `moc foo.hpp` produces moc_foo.cpp
  `moc foo.cpp` produces foo.moc (a C++ source file)
  `moc foo.mm`  produces foo.moc (note: also a C++ source file)

Most moc outputs are included, but some are compiled.

- Find included header moc outputs:

    grep -REh 'include.+moc_.+\.cpp' upstream/qtbase/src/<module>

- Find included source file moc outputs:

    grep -REh 'include.+\.moc' upstream/qtbase/src/<module>

- Find compiled moc outputs (from within the upstream build directory):

    find . -type f -name mocs_compilation.cpp -exec grep -h '#include' {} \;

2. TODO upon upstream upgrade

2.1. A note on Qt Long Term Support (LTS) versions

Some minor versions are chosen to become LTS releases. They become LTS when
the next minor version is released. For example, at the time that 6.6.0 was
released, the latest 6.5 release was 6.5.3, which meant that 6.5 became "6.5
LTS" with the 6.5.4 release.

However, LTS releases are only available to commercial license holders. The
git repository accessible to Open Source users contains only the non-LTS
release branches. Therefore the LTS status of a particular minor version is
irrelevant to Open Source users and they should always upgrade to the latest
minor version.

2.2. Upgrade upstream/qtbase/ to the latest release.

2.3. Check whether upstream versions of any files in downstream/ have changed.

Print all files under downstream/ that have changed upstream since the last
release (using version 6.4.1 in this example):

  $ ./find-outdated-patches v6.4.1
  qtbase/src/tools/moc/generator.cpp
  qtbase/src/tools/moc/main.cpp
  qtbase/src/corelib/thread/qmutex.cpp
  qtbase/src/corelib/compat/removed_api.cpp
  qtbase/src/corelib/serialization/qcborstreamreader.cpp

Then make use of something like `git add --patch` to merge the new upstream
changes and our downstream patches:

  $ cp {upstream,downstream}/qtbase/src/tools/moc/generator.cpp

  Use the latest upstream patch to differentiate new upstream code from our
  patch code:

  $ git -C upstream/qtbase diff v6.4.1 -- src/tools/moc/generator.cpp

  Merge the upstream patch into our patched file:

  $ git add --patch downstream/qtbase/src/tools/moc/generator.cpp

2.4. Check whether any bugs reported upstream have been resolved.

Upstream bugs usually require downstream patches so check whether patches for
resolved bugs can be removed.

Note: Open upstream bugs should be recorded somewhere in this file.

2.5. Run the generate script.

2.6. Ensure resource collection files (.qrc) have not changed.

In the upstream build all of the .qrc files are generated by cmake. We use
copies of the upstream-generated ones but usually they need to be edited (for
example, to make absolute paths relative).

It's probably easiest to look for changes to any calls to
qt_internal_add_resource() -- the function used to define a .qrc to be
generated -- in each module's CMakeLists.txt (if only because not all .qrc
files are generated on all platforms).

Otherwise, to generate the .qrc files, simply run cmake on the upstream build:

  $ mkdir qt6-build
  $ cd qt6-build
  $ cmake ~/work/build2-packaging/Qt6/upstream/qtbase
  $ find . -name \*.qrc
  ./src/gui/.rcc/qpdf.qrc
  ./src/gui/.rcc/gui_shaders.qrc
  ./src/widgets/.rcc/qstyle1.qrc
  ./src/widgets/.rcc/qstyle.qrc
  ./src/widgets/.rcc/qmessagebox.qrc

2.7. Check for new or removed Qt configuration features and associated macros

Run the `check-config-header` script for all modules:

./check-config-header --mod global  --visibility pub
./check-config-header --mod global  --visibility priv
./check-config-header --mod core    --visibility pub
./check-config-header --mod core    --visibility priv
./check-config-header --mod gui     --visibility pub
./check-config-header --mod gui     --visibility priv
./check-config-header --mod widgets --visibility pub
./check-config-header --mod widgets --visibility priv

This script prints the names of features and macros that are not handled both
upstream and downstream, so this list will include features and macros added
or removed by the upstream upgrade. Update the downstream module config
headers indicated by the module and visibility options passed as required (see
the script itself for the list of headers).

2.8. Update each module's build

To see changes in a module's build (QtCore, in this example):

  $ git -C upstream/qtbase diff v6.6.1  -- src/corelib/CMakeLists.txt

Look out for dependency changes, new/removed source files, and added/removed
macro definitions. Also check that we're not still moc'ing sources that no
longer need it.

2.9. Look for changes to common macros

Search the changes in upstream/qtbase/cmake/ and upstream/qtbase/.cmake.conf
for instances of the macro-defining functions and variables (listed above in
the "Upstream build documentation" section):

  $ git -C upstream/qtbase diff v6.6.1  -- cmake/

2.10. Confirm we got all of the macros right

Compare the macros defined in our build to that of the upstream build. This is
the only way to be sure we got them right because the upstream build defines
macros in multiple places using multiple methods. (See Appendix A for upstream
build instructions.)

2.11. Misc.

- Enable OpenSSL 3 support when it's been packaged for build2. (See
  QT_FEATURE_openssl and QT_FEATURE_opensslv30 in qconfig.h.in.)

- Consider linking Qt6Rcc and Qt6Uic against the proper libQt6Core. Currently,
  in order to avoid having to build two copies of libQt6Core (the second one
  being in the host configuration), Qt6{Moc,Rcc,Uic} get a reduced set of
  QtCore sources (approximating the upstream bootstrap library) compiled in
  instead. In addition to the fact that we're compiling the same sources three
  times, upstream is trending towards using more and more QtCore functionality
  so this might not remain practical for long.
  
3. Platform/compiler support issues

Mac OS with GCC is not supported because certain system headers make use of
Objective-C blocks
(https://en.wikipedia.org/wiki/Blocks_(C_language_extension)) which are not
supported by GNU gcc/g++. There is an open issue to add support
(https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78352) but it's stalled for what
looks like licensing reasons (see comment 16).

libQt6Gui and libQt6Widgets are not supported on FreeBSD because those CI VMs
do not have the X Window System installed.

4. Upstream bugs

None, currently.

Appendix A: Running the upstream build

Upstream reference: https://doc.qt.io/qt-6/build-sources.html

A.1. Windows

Upstream reference: https://doc.qt.io/qt-6/windows-building.html

These instructions work on our Windows CI VMs.

Begin from C:\tmp:

  cd C:\tmp

Make various tools available:

  set PATH=%PATH%;C:\build2\bin

Download and install cmake:

  Find download link at https://github.com/Kitware/CMake/releases/.

  curl -L -A git/2.42.0 -sS -O <cmake-download-URL>

  - Run the graphical installer, installing into C:\tmp\cmake.

  - Append cmake.exe location to PATH:

    set PATH=%PATH%;C:\tmp\cmake\bin

Download and "install" Ninja:

  Find download link at https://github.com/ninja-build/ninja/releases/.

  curl -L -A git/2.42.0 -sS -O <ninja-download-URL>
  unzip ninja-win.zip
  move ninja.exe C:\build2\bin

Download the python installer:

  Find download link at https://www.python.org/downloads/windows/.

  curl -L -A git/2.42.0 -sS -O <python-download-URL>

  - Run graphic installer, installing into C:\tmp\python.

  - Append python.exe location to PATH:

    set PATH=%PATH%;C:\tmp\python

Set up VS environment:

  "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64

Clone the upstream Qt repository:

  git clone git://code.qt.io/qt/qtbase.git

Check out the version we're working on:

  git -C qtbase checkout v6.7.2

Set up the Qt build with verbose output:

  mkdir qtbuild
  cd qtbuild
  ..\qtbase\configure
  cmake -DCMAKE_VERBOSE_MAKEFILE=1 ..\qtbase

Run the build, sending output to a file:

  cmake --build . > C:\tmp\qtbuild.txt

(To clean: cmake --build . --target clean)

Download the build log from dev machine:

  scp -o PreferredAuthentications=password \
      -o UserKnownHostsFile=/dev/null \
      -o StrictHostKeyChecking=no \
      -o LogLevel=ERROR \
      -P <ssh-tunnel-port> \
      build@localhost:C:/tmp/qtbuild.txt \
      ~

A.2. macOS

Upstream reference: https://doc.qt.io/qt-6/macos-building.html

These instructions work on our macOS CI VMs.

Download and install cmake:

  Find download link at https://github.com/Kitware/CMake/releases/.

  curl -L -A git/2.42.0 -sS -O <cmake-download-URL>

  Run the graphical installer:

    open cmake-3.30.0-macos10.10-universal.dmg

  Create the cmake executable symlinks:

    sudo "/Applications/CMake.app/Contents/bin/cmake-gui" --install

Download and "install" Ninja:

  Find download link at https://github.com/ninja-build/ninja/releases/.

  curl -L -A git/2.42.0 -sS -O <ninja-download-URL>
  unzip ninja-mac.zip
  sudo ln -s $PWD/ninja /usr/local/bin

Clone the upstream Qt repository:

  git clone git://code.qt.io/qt/qtbase.git

Check out the version we're working on:

  git -C qtbase checkout v6.7.2

Set up the Qt build with verbose output:

  mkdir qtbuild
  cd qtbuild
  ../qtbase/configure
  cmake -DCMAKE_VERBOSE_MAKEFILE=1 ../qtbase

Run the build, sending output to a file:

  cmake --build . > qtbuild.txt

(To clean: cmake --build . --target clean)

Create an archive containing the cmake autogen (moc, rcc, uic) build
parameters:

  tar czf moc_mac.tgz src/corelib/CMakeFiles/Core_autogen.dir \
                      src/gui/CMakeFiles/Gui_autogen.dir \
                      src/widgets/CMakeFiles/Widgets_autogen.dir

Download the build log from dev machine:

  scp -o PreferredAuthentications=password \
      -o UserKnownHostsFile=/dev/null \
      -o StrictHostKeyChecking=no \
      -o LogLevel=ERROR \
      -P <ssh-tunnel-port> \
      build@localhost:qtbuild.txt \
      ~

Download the cmake autogen archive from dev machine:

  scp -o PreferredAuthentications=password \
      -o UserKnownHostsFile=/dev/null \
      -o StrictHostKeyChecking=no \
      -o LogLevel=ERROR \
      -P <ssh-tunnel-port> \
      build@localhost:qtbuild/moc_mac.tgz \
      ~