Clarifying current status of LTO (b_lto) support

[GertyP]

I've been trying to clarify the status of meson + lto functionality, using clang, specifically the msvc-style clang (i.e. 'clang-cl') and from this comment from @dcbaker -

It looks like there's (as always) something not working correctly on windows

it seems like you might be able to confirm or correct my understanding of the current state of things, which is not obvious from simply trying to set a meson project up with default_options: ['b_lto=true', ...] -

1. On windows, when using the msvc-style clang compiler/linker tools ('clang-cl.exe' and 'lld-link.exe'), this gives a meson compiler id of 'clang-cl'. The 'allplatformtest.py', 'test_static_library_lto' test is doing a check to skip this test, using only if detect_c_compiler(env, MachineChoice.HOST).get_id() == 'clang' and is_windows(): ... which is checking for use of the gnu-style clang.exe compiler (ID = 'clang'); it doesn't catch the msvc-style 'clang-cl.exe' (ID = 'clang-cl'). However, when the test proceeds to perform the meson setup -Db_lto=true ..., all settings related to use of lto (i.e. -flto compile options) are quietly ignored/avoided and the project compiles and links without any LTO and the test succeeds. Is all of this expected?

2. If, again on windows, I explicitly point to the gnu-style clang compiler ('clang.exe'), and attempt to 'meson setup and compile' this same '5 linkstatic' test (doing the following in the 'x64 Native tools command prompt for VS 2022') -

set CC=clang
set CXX=clang
set CC_LD=lld-lin
set CXX_LD=lld-link
python.exe meson.py setup --backend ninja -Db_lto=true meson_lto_test_builddir "c:\dev\meson\test cases\common\5 linkstatic"
python.exe meson.py compile -v -C meson_lto_test_builddir

then I get the following successful steps (I'm trimming a little of the output for compactness) -

[1/7] clang "-I..\..\dev\meson\test cases\common\5 linkstatic" -flto -D_FILE_OFFSET_BITS=64 -O0 -o libmylib.a.p/libfile.c.obj -c ../../dev/meson/test cases/common/5 linkstatic/libfile.c

[2/7] clang "-I..\..\dev\meson\test cases\common\5 linkstatic" -flto -D_FILE_OFFSET_BITS=64 -O0 -o libmylib.a.p/libfile2.c.obj -c ../../dev/meson/test cases/common/5 linkstatic/libfile2.c

[3/7] clang "-I..\..\dev\meson\test cases\common\5 linkstatic" -flto -D_FILE_OFFSET_BITS=64 -O0 -o libmylib.a.p/libfile3.c.obj -c ../../dev/meson/test cases/common/5 linkstatic/libfile3.c

[4/7] clang "-I..\..\dev\meson\test cases\common\5 linkstatic" -flto -D_FILE_OFFSET_BITS=64 -O0 -o libmylib.a.p/libfile4.c.obj -c
../../dev/meson/test cases/common/5 linkstatic/libfile4.c

[5/7] clang "-I..\..\dev\meson\test cases\common\5 linkstatic"-flto -D_FILE_OFFSET_BITS=64 -O0 -o prog.exe.p/main.c.obj -c ../../dev/meson/test cases/common/5 linkstatic/main.c

[6/7] llvm-ar csrDT libmylib.a libmylib.a.p/libfile.c.obj libmylib.a.p/libfile2.c.obj libmylib.a.p/libfile3.c.obj libmylib.a.p/libfile4.c.obj

[7/7] clang -Wl,/OUT:prog.exe prog.exe.p/main.c.obj -flto -fuse-ld=lld-link -Wl,/DEBUG libmylib.a -Wl,/SUBSYSTEM:CONSOLE

Similarly, if I point the compiler to the msvc-style 'clang-cl' (set CC=clang-cl, set CXX=clang-cl) and repeat the meson setup and compile steps, it all builds successfully ... only this time, I can see that it doesn't use any -flto options anywhere

Is this expected behaviour, for the clang-cl compiler to quietly ignore the user's request to use 'b_lto=true'?

3. If I manually add the -flto switch (it only seems to be needed for the compile steps) and run run the compile (clang-cl.exe -flto ...), lib (llvm-ar ...), and link (lld-link ...) steps, using the msvc-style clang compiler windows tools... it all works fine.

Is that expected?

What is the windows-related problem you're referring to in the quote above, and is there an open issue for it and any progress on resolving it?

4. I do notice that it's critical to also set the linker environment variable (CC_LD=lld-link, CXX_LD=lld-link), without which, we get a link error (routed through clang.exe ... instead of lld-link.exe ...), complaining of corruption in 'main.c.obj', so I wonder if that was what was giving you problems. Is there a way for meson to identify the scenario where the user specifies the clang compiler ... but NOT the 'lld-link' linker, since that scenario gives us problems and would perhaps better be dealt with as an error message (if not a more specific skip-check) for this test? If feels like this should actually be a check and error in meson's internals, not just a skip-check for this particular test. What do you think?

5. Since gnu-style clang tools on windows correctly configures and runs the '5 linkstatic' test with 'b_lto=true', all without error, what is preventing the 'test_static_library_lto' (and similar tests) from removing that "windows and id=='clang'"-skip check?

6. The same goes for use of the msvc-style clang-cl compiler windows tools; manually running these compile, lib, and link steps -

clang-cl "/I..\..\dev\meson\test cases\common\5 linkstatic" -flto -D_FILE_OFFSET_BITS=64 /Wall -Od /Zi /Folibfile.c.obj /c "../../dev/meson/test cases/common/5 linkstatic/libfile.c"
clang-cl ... -flto ... libfile2.c
clang-cl ... -flto ... libfile3.c
clang-cl ... -flto ... libfile4.c
llvm-lib /machine:x64 /out:libmylib.lib libfile.c.obj libfile2.c.obj libfile3.c.obj libfile4.c.obj
lld-link /machine:x64 /out:prog.exe /subsystem:console /defaultlib:libcmt.lib libmylib.lib main.c.obj

all works fine (again, provided 'lld-link' is used), so what is preventing meson's full support of 'b_lto=true', when using 'clang-cl' on windows?

[GertyP]

After having poked around with the code to get answers for myself -

1. It's probably not expected; probably also meant to check for compiler id 'clang-cl', too. But either way the internal compiler clang/clang-cl classes on windows did not enable the LTO options, so they're quietly ignored.
2. Yes; the lto options needed to be explicitly enabled per Compiler class.
3. Don't know about anyone's expectations before, but manually adding the LTO options to the compile, lib/archive, and link args for those tools all belonging to the same group (i.e. LLVM tools), then it all works as advertised.
4. I think only @dcbaker knows what the problem being referred to by "It looks like there's (as always) something not working correctly on windows" really was. The mixing of tools from different families (e.g. llvm vs msvc) is something meson seems keen to do, and was the only real problem I could find.
5. (and 6) Just enabling support for meson's LTO options in the relevant compiler and linker classes is all that's really needed, but I've added this and a few other changes in this PR.