MOC_LVL_0, arguments cleanup, out of source builds, etc.

Added support for MOC_LVL_0
Added support for basic/shadow builds (and modified out of source builds)
Modified default path of UNITY_TMP_DIR, it is now in the debug or shadow_build_debug. That being said, in regular (non shadow) mode, this modifies QtCreator display behaviour (2 entries are displayed, one for debug and one for release)
Improved cli arguments parsing
Added strategy configuration (UNITY_STRATEGY)
Removed one process step from qmake_unity.pri (see build_pass).

Pragma once were added in test projects to test MOC_LVL_2) (using guard2once)

Author: nmoreaud

.gitignore

@@ -33,8 +33,8 @@
 
 # Qt
 *_plugin_import.cpp
-/.qmake.cache
-/.qmake.stash
+.qmake.cache
+.qmake.stash
 *.pro.user
 *.pro.user.*
 *.qbs.user
@@ -48,6 +48,10 @@ ui_*.h
 *.jsc
 Makefile*
 *build-*
+debug/
+release/
+object_script.*.Debug
+object_script.*.Release
 
 # Python 
 __pycache__/
@@ -56,4 +60,5 @@ __pycache__/
 
 # Perso
 
-perso.txt
+perso.txt
+tmp/

README.md

@@ -41,8 +41,16 @@ include(path/to/qmakeUnity/qmake_unity.pri)
 You might want to adjust the following variables in your .pro file before including qmake_unity.pri : 
 
 - `UNITY_TMP_DIR`
-Choose a path where cpp/moc/obj files generated by qmake-unity will be stored
-Default value : `qmake-unity.pri's folder/../tmp/Build_Qt/unity`
+Choose a path where temporary cpp/h/pri files generated by qmake-unity will be stored.
+Default value : `BuildPwd/unity/`
+
+Please read `qmake_unity.pri` if you want to put move generated/object files out of source.
+
+- `UNITY_STRATEGY`
+There are 3 possible strategies to group cpp files together : 
+- `incremental` is the default mode. The group size depends on the project's number of cpp files. This option is recommended for big projects.
+- `per-processor` generates one CPP per processor. You can set your number of processors in `unity_config.py`.
+- `single-compilation-unit` : generated a single CPP file for your entire project.
 
 - `UNITY_MOC_MODE`
 `MOC_LVL_1` : group the compilation of moc_file1.cpp and moc_file2.cpp
@@ -109,7 +117,7 @@ b.cpp
 QString status(const Screen &screen, const Display &display) { return display.status(screen); }
 ```
 
-**MACRO with side effects**
+**One macro has a side effect that triggers compile errors**
 ```
 a.h
 #define Status(XXX, YYY) XXX+YYY-2
@@ -118,16 +126,18 @@ b.cpp (doesn't include a.h)
 void displayStatus(const Screen &screen, const Display &display) {}
 ```
 a.cpp may be grouped with b.cpp though they are not related. The macro `Status` has a very short and common name that will match `displayStatus` and replace it with something irrelevant.
+Please choose a longer and unique macro name
 
 **Header files that must be included in a certain order ; or that may cause conflicts.**
 
 That is not a good practice, but sometimes things are like that.
 For example, some windows SDK headers must be included in a certain order.
 In that case, you might remove the header from unity build by adding `//@NO_UNITY` to the including header.
 
-**There is an error related with unity builds, but I don't know which file causes it**
+**There is an error related with unity builds, but I don't know which file causes it.**
+
 If you don't know which files causes an error, you can either : 
-  - Edit the `tmp/unity/XXX/unity_[0123].cpp` files and comment entries until the error disappears. Then you'll know which file raises the problem.
+  - Edit the `tmp/unity/unity_[0123].cpp` files and comment entries until the error disappears. Then you'll know which file raises the problem.
   - Edit your .pro file to include qmake_unity.pro when the `SOURCES` and `HEADERS` variables are just partially filled, then run qmake.
 
 **There is an error on my build server/my colleague's workstation but it builds fine on my computer. How can I fix it ?**
@@ -136,7 +146,7 @@ The sources files are probably not be grouped identically on your computer and t
 For example, it happens when you add and remove files from a project.
 Procedure to reproduce : 
 - build the project locally
-- copy and paste the contents of your colleague's `tmp/unity/XXX` directory in you own tmp directory
+- copy and paste the contents of your colleague's `tmp/unity/` directory in you own tmp directory
 - build the project without calling qmake (make only) ; the same error should happen.
 
 **The `__FILE__` macro doesn't return the filename anymore, it return a relative path !**
@@ -163,13 +173,14 @@ QString unityExtractFileNameFromPath(const char * iFilePath)
 * `MOC_LVL_2` works only with pragma once; there is no workaround for now
 * Visual studio IDE is not supported : the source files are be displayed twice.
 Either build in QtCreator or from command line.
+* When running qmake, qmake-unity logs appear twice. This is because qmake processes every pro file 3 times, one for Makefile, one for Makefile.debug and another one for Makefile.release
 
 ## What else can I do to speedup my build
 
 **Pure C++ techniques :**
 - Forward declare headers (incremental build)
 - Write less templates
-- Use precompiled headers
+- Use precompiled headers (PCH)
 
 **Qt :**
 - If you build in VisualStudio, install Qt Visual Studio Tools <= 2.2; which now supports parallel calls of moc.
@@ -187,7 +198,7 @@ Either build in QtCreator or from command line.
 
 **Use better tools :**
 - Zapcc is a fast alternative to Clang
-- gold linker replaces ld (linux)
+- Gold linker replaces ld (linux)
 - Replace Make with Ninja (best combo with CMake)
 - Add COTIRE to CMake to handle unity builds and precompiled headers
 - Install Incredybuild on Windows

qmakeUnity/qmake_unity.pri

@@ -1,27 +1,57 @@
-UNITY_SCRIPTS_DIR = $$clean_path($${PWD})
+!equals(UNITY_BUILD_DISABLE, "1") {
 
-!defined(PROJECTS_ROOT, var) {
-    PROJECTS_ROOT=$$clean_path($${PWD}/../)
-}
-!defined(UNITY_TMP_DIR, var) {
-    UNITY_TMP_DIR=$$clean_path($$PROJECTS_ROOT/tmp/Build_Qt/unity/)
-}
-UNITY_TMP_DIR_SUFFIX = $$relative_path($$_PRO_FILE_PWD_, $$PROJECTS_ROOT)
-UNITY_DIR = $$UNITY_TMP_DIR/$$UNITY_TMP_DIR_SUFFIX/
+    build_pass { # do this action once for debug and once for release
 
+        UNITY_SCRIPTS_DIR = $$clean_path($${PWD})
 
-!defined(UNITY_MOC_MODE, var) {
-    UNITY_MOC_MODE = MOC_LVL_2
-}
+        !defined(UNITY_DIR, var) {
+            # This config makes QtCreator display both release and debug unity directory in "project view" > "source tree"
+            # when shadow build are disabled.
+            # If this annoys you, you can enable out of source builds, please see bellow)
+            CONFIG(debug, debug|release) {
+                UNITY_DIR = $$OUT_PWD/debug/unity/
+            } else {
+                UNITY_DIR = $$OUT_PWD/release/unity/
+            }
+        }
+
+        # If you have many projects, you might want to put unity and other tmp files outside of the hierarchy. (out of source builds)
+        # To do this, uncomment the following lines
+        #
+        # PROJECTS_ROOT is the path to the "src" directory
+        # PROJECTS_ROOT = $$clean_path($${PWD}/../)
+        # PROJECT_NAME = $$relative_path($$_PRO_FILE_PWD_, $$PROJECTS_ROOT)
+        # OUTPUT_BASE_PATH = $$PROJECTS_ROOT/tmp/Build_Qt/$$PROJECT_NAME
+        #
+        # CONFIG( debug, debug|release ) {
+        #     OUTPUT_BASE_PATH = $$OUTPUT_BASE_PATH/debug
+        # } else {
+        #     OUTPUT_BASE_PATH = $$OUTPUT_BASE_PATH/release
+        # }
+        # UNITY_TMP_DIR = $$OUTPUT_BASE_PATH/unity/
+        #
+        # To also put the object files outside, uncomment the following lines
+        #
+        # OBJECTS_DIR = $$OUTPUT_BASE_PATH/obj/
+        # MOC_DIR = $$OUTPUT_BASE_PATH/moc/
+        # UI_DIR = $$OUTPUT_BASE_PATH/ui/
+        # RCC_DIR = $$OUTPUT_BASE_PATH/rcc/
+        # PRECOMPILED_DIR = $$OUTPUT_BASE_PATH/pch/
+
+        !defined(UNITY_MOC_MODE, var) {
+            UNITY_MOC_MODE = MOC_LVL_2
+        }
+
+        !defined(UNITY_STRATEGY, var) {
+            UNITY_STRATEGY = incremental
+        }
 
-#!build_pass { # do this action only once if config = debug + release
-    !equals(UNITY_BUILD_DISABLE, "1") {
         defined(UNITY_BUILD, var) {
             !count(SOURCES, 1) {
                 mkpath($$UNITY_DIR)
                 UNITY_SOURCES_FILE_LOCALVAR = $${UNITY_DIR}unitySources.txt
                 write_file($$UNITY_SOURCES_FILE_LOCALVAR, SOURCES)
-                system(cd $$_PRO_FILE_PWD_ & python $$UNITY_SCRIPTS_DIR/qmake_unity.py update incremental "$$UNITY_DIR" "$$UNITY_SOURCES_FILE_LOCALVAR")
+                system(cd $$_PRO_FILE_PWD_ & python $$UNITY_SCRIPTS_DIR/qmake_unity.py --mode update --strategy $$UNITY_STRATEGY --mocMode $$UNITY_MOC_MODE --tmpDir "$$UNITY_DIR" --sourceListPath "$$UNITY_SOURCES_FILE_LOCALVAR")
                 UNITY_GENERATED_PRI_LINES = $$cat($${UNITY_DIR}unity.pri, lines)
                 for(UNITY_GENERATED_PRI_LINE, UNITY_GENERATED_PRI_LINES) {
                     eval($$UNITY_GENERATED_PRI_LINE)
@@ -33,7 +63,7 @@ UNITY_DIR = $$UNITY_TMP_DIR/$$UNITY_TMP_DIR_SUFFIX/
                     mkpath($$UNITY_DIR)
                     UNITY_HEADERS_FILE_LOCALVAR = $${UNITY_DIR}unityHeaders.txt
                     write_file($$UNITY_HEADERS_FILE_LOCALVAR, HEADERS)
-                    system(cd $$_PRO_FILE_PWD_ & python $$UNITY_SCRIPTS_DIR/unity_moc_headers.py generate_groups "$$UNITY_HEADERS_FILE_LOCALVAR")
+                    system(cd $$_PRO_FILE_PWD_ & python $$UNITY_SCRIPTS_DIR/unity_moc_headers.py --mode generate_groups --headerListPath "$$UNITY_HEADERS_FILE_LOCALVAR")
                     UNITY_GENERATED_PRI_LINES = $$cat($${UNITY_DIR}unity_headers.pri, lines)
                     for(UNITY_GENERATED_PRI_LINE, UNITY_GENERATED_PRI_LINES) {
                         eval($$UNITY_GENERATED_PRI_LINE)
@@ -42,7 +72,7 @@ UNITY_DIR = $$UNITY_TMP_DIR/$$UNITY_TMP_DIR_SUFFIX/
             }
         }
     }
-    else {
-        !build_pass:message("UNITY_BUILD_DISABLE is set.")
-    }
-#}
+}
+else {
+    !build_pass:message("UNITY_BUILD_DISABLE is set.")
+}

qmakeUnity/qmake_unity.py

@@ -1,5 +1,6 @@
 #!/usr/bin/env python3
 
+import argparse
 import glob
 import os
 import re
@@ -61,7 +62,7 @@ def writeToDisk(self, type, unityDirectory):
         fileContent += "\n".join(fileIncludes)
         fileContent += "\n"
 
-        # on n'écrit que si le fichier a été modifié
+        # we write only if the content has been modified
         fileOldContent = ''
         try:
             with open(self.file, encoding="utf-8") as file:
@@ -147,7 +148,7 @@ def addNewSourceFiles(groupList, sourceList, type, desiredGroupsSize):
             if j % desiredGroupsSize == 0:
                 g = Group()
                 g.type = type
-                # crée une copie de la liste
+                # creates a copy of the list
                 g.sources = list(sources)
                 groupList.append(g)
                 sources.clear()
@@ -254,6 +255,7 @@ def writePriFile(groupList, unityPriFile, type, priWriteMode):
             groupFiles = [group.file for group in groupList]
             print("SOURCES += " + " ".join(groupFiles), file=file)
 
+            # To remove a cpp file from qmake, you must type the exact same path as qmake passed it to this script through unitySources.txt
             if type == 'cpp':
                 sourceFiles = [source.pathFromProject for group in groupList for source in group.sources]
                 print("SOURCES -= " + " ".join(sourceFiles), file=file)
@@ -343,9 +345,9 @@ def removeIncompatibleSourcesFromList(cppList):
 
 
 def argumentsCheck(mode, unityDirectory, strategy, cppList):
-    assert mode in ["update","clear","removeOldSources","qmakeListRemovedSources","qmakeListAddedSources"], "Incorrect mode : " + mode
+    assert mode in ["update","clear"], "Incorrect mode : " + mode
     if (mode != "clear"):
-        assert strategy in ["incremental","per-processor","single"], "Stratégie incorrecte : " + strategy
+        assert strategy in ["incremental","per-processor","single-compilation-unit"], "Unknown strategy : " + strategy
     assert "unity" in unityDirectory, "The folder " + unityDirectory + " must contain the keyword 'unity'"
     assert os.path.exists(unityDirectory), "The folder " + unityDirectory + " doesn't exists. This error is probably the consequence of another error"
     cd = os.getcwd()
@@ -354,27 +356,32 @@ def argumentsCheck(mode, unityDirectory, strategy, cppList):
     for cpp in cppList:
         assert cpp.pathFromProject.endswith(".cpp") or cpp.pathFromProject.endswith('.c'), "Error : this file is not a CPP : " + cpp
 
+def buildArgsParser():
+    parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument('--mode', type=str, choices=('update', 'clear'))
+    parser.add_argument('--strategy', default='incremental', choices=('incremental', 'per-processor', 'single-compilation-unit'),
+                        help='Strategy for grouping files together')
+    parser.add_argument('--tmpDir', type=str)
+    parser.add_argument('--mocMode', type=str, choices=('MOC_LVL_0', 'MOC_LVL_1', 'MOC_LVL_2'), default='MOC_LVL_1')
+    parser.add_argument('--sourceListPath', type=str)
 
+    return parser
+    
 def main():
-    if len(sys.argv) <= 3:
-        print("""Usage : python qmake-unity.py mode strategy path_to_unity.pri [source1.cpp, subdir/source2.cpp ...]
-Mode : update/clear/removeOldSources/QMAKE_LIST_REMOVED/QMAKE_LIST_ADDED
-Strategy: incremental(little groups)/per-processor(one group per processor)/unity(one group)
-""")
-        return
-
     try:
-        #print_debug("current dir : " + os.getcwd())
-        mode = sys.argv[1]
-        strategy = sys.argv[2]
-        unityDirectory = os.path.normpath(sys.argv[3])
+        # Must be executed from the project's source directory (which contains the .pro file)
+        # print_dev("current dir : " + os.getcwd())
+        cliArgs = buildArgsParser().parse_args()
+        
+        mode = cliArgs.mode
+        strategy = cliArgs.strategy
+        unityDirectory = os.path.normpath(cliArgs.tmpDir)
         unityPriFile = os.path.normpath(unityDirectory + "/unity.pri")
-        #print_debug(unityDirectory)
-
+        mocMode = cliArgs.mocMode
+        inputProjectCppListFile = cliArgs.sourceListPath
 
         projectCppListPath = []
-        if len(sys.argv) > 4:
-            inputProjectCppListFile = sys.argv[4]
+        if inputProjectCppListFile is not None:
             assert os.path.exists(inputProjectCppListFile), "The sources file list doesn't exists : " + inputProjectCppListFile + ". This error is probably the consequence of another error."
 
             with open(inputProjectCppListFile) as file:
@@ -416,9 +423,16 @@ def main():
             cppGroupSize = len(cppList)
 
         generateUnityBuildFiles(unityPriFile, unityDirectory, mode, 'cpp', cppGroupList, cppList, cppGroupSize, "w+")
-        print_debug('todo : decomment only for MOC_LVL_1')
-        print_debug('todo : comment only for MOC_LVL_0 and MOC_LVL_2')
-        generateUnityBuildFiles(unityPriFile, unityDirectory, mode, 'moc', mocGroupList, mocList, MOC_GROUPSIZE, "a")
+        
+        if mocMode == 'MOC_LVL_1':
+            # LVL_0 = no moc optimization
+            # LVL_1 = moc files, the group them for CL step
+            # LVL_2 = group moc calls before the CL step ; see unity_moc_headers
+            generateUnityBuildFiles(unityPriFile, unityDirectory, mode, 'moc', mocGroupList, mocList, MOC_GROUPSIZE, "a")
+        else:
+            for group in mocGroupList:
+                group.removeFromDisk()
+
 
     except InlineException as err:
         sys.stderr.write(err.message + "\n")

qmakeUnity/unity_common.py

@@ -23,7 +23,7 @@ def utf8_file_read(file):
 
 def file_remove(filePath):
     if SAFE_MODE:
-        print_debug("SAFE MODE : simulate os.remove(" + filePath + ")", True)
+        print_debug(f'SAFE MODE - simulate delete file "{filePath}" ({os.path.abspath(filePath)})', True)
     else:
         os.remove(filePath)
 

qmakeUnity/unity_config.py

@@ -5,10 +5,11 @@
 LOG_DEBUG = True
 
 # List of namespace names that are allowed appear in a "using namespace XXX" expression in a group
-NAMESPACE_WHITELIST = ["GoodNamespace"]
+# ex : NAMESPACE_WHITELIST = ["good::namespace"]
+NAMESPACE_WHITELIST = []
 
 # we let 1 more to avoid freezes
 NB_PROCESSORS = 7 
 
-# nb files that will be mocced together
+# nb files that will be mocced together (MOC_LVL_1 or MOC_LVL_2)
 MOC_GROUPSIZE = 50