[toolchains] allowlist_include_directories does not work on non-absolute paths

[armandomontanez]

A while back while rolling through Bazel updates I noticed changes in behaviors to cxx_builtin_include_directories (allowlist_include_directories in rule-based toolchains) where it seemed to stop properly allowlisting toolchain builtin include paths.
Inspired by bazelbuild/bazel#23615, I wanted to create a minimal reproducer, which is pretty easy:

diff --git a/examples/rule_based_toolchain/toolchain/BUILD.bazel b/examples/rule_based_toolchain/toolchain/BUILD.bazel
index 58e3540..b95f795 100644
--- a/examples/rule_based_toolchain/toolchain/BUILD.bazel
+++ b/examples/rule_based_toolchain/toolchain/BUILD.bazel
@@ -24,7 +24,6 @@ cc_toolchain(
         ],
         "//conditions:default": [],
     }) + [
-        "//toolchain/args:no_canonical_prefixes",
         "//toolchain/args:warnings",
     ],
     enabled_features = ["@rules_cc//cc/toolchains/args:experimental_replace_legacy_action_config_features"],
diff --git a/examples/rule_based_toolchain/toolchain/tools/BUILD.bazel b/examples/rule_based_toolchain/toolchain/tools/BUILD.bazel
index ccd0060..9f57c0c 100644
--- a/examples/rule_based_toolchain/toolchain/tools/BUILD.bazel
+++ b/examples/rule_based_toolchain/toolchain/tools/BUILD.bazel
@@ -64,6 +64,16 @@ cc_tool(
         ":exec_platform_builtin_headers",
         ":exec_platform_multicall_support_files",
     ],
+    allowlist_include_directories = [
+        "@linux_sysroot//:usr-include-x86_64-linux-gnu",
+    ] + select({
+        "//constraint:linux_aarch64": [
+            "@clang-linux-aarch64//:lib-clang-include",
+        ],
+        "//constraint:linux_x86_64": [
+            "@clang-linux-x86_64//:lib-clang-include",
+        ],
+    }),
 )

 cc_tool(
@@ -76,6 +86,16 @@ cc_tool(
         ":exec_platform_builtin_headers",
         ":exec_platform_multicall_support_files",
     ],
+    allowlist_include_directories = select({
+        "//constraint:linux_aarch64": [
+            "@clang-linux-aarch64//:include-c++-v1",
+            "@clang-linux-aarch64//:include-x86_64-unknown-linux-gnu-c++-v1",
+        ],
+        "//constraint:linux_x86_64": [
+            "@clang-linux-x86_64//:include-c++-v1",
+            "@clang-linux-x86_64//:include-x86_64-unknown-linux-gnu-c++-v1",
+        ],
+    }),
 )

 cc_tool(

This fails as seen below:

[...]/rules_cc/examples/rule_based_toolchain$ bazel test //...
INFO: Analyzed 42 targets (0 packages loaded, 29 targets configured).
ERROR: /usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/e38ec8e746bf3d11672a536413f07932/external/googletest~/BUILD.bazel:86:11: Compiling googletest/src/gtest-filepath.cc failed: absolute path inclusion(s) found in rule '@@googletest~//:gtest':
the source file 'googletest/src/gtest-filepath.cc' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain):
  '/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/e38ec8e746bf3d11672a536413f07932/external/_main~_repo_rules~clang-linux-x86_64/include/c++/v1/string'
  '/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/e38ec8e746bf3d11672a536413f07932/external/_main~_repo_rules~clang-linux-x86_64/include/c++/v1/__algorithm/max.h'
  '/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/e38ec8e746bf3d11672a536413f07932/external/_main~_repo_rules~clang-linux-x86_64/include/c++/v1/__algorithm/comp.h'
  '/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/e38ec8e746bf3d11672a536413f07932/external/_main~_repo_rules~clang-linux-x86_64/include/c++/v1/__config'
  '/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/e38ec8e746bf3d11672a536413f07932/external/_main~_repo_rules~clang-linux-x86_64/include/x86_64-unknown-linux-gnu/c++/v1/__config_site'
 [many more]

[armandomontanez]

Memory is getting jogged... I think this broke when we migrated to bzlmod.

[armandomontanez]

It looks like this is rooted in issues where the relative paths for directories end up once in the sandbox:

Path '/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/e38ec8e746bf3d11672a536413f07932/external/+_repo_rules+clang-linux-x86_64/include/c++/v1/memory' is not under '/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/e38ec8e746bf3d11672a536413f07932/execroot/_main', cannot relativize

Shortened:

- e38ec8e746bf3d11672a536413f07932/external/+_repo_rules+clang-linux-x86_64/include/c++/v1/memory
+ e38ec8e746bf3d11672a536413f07932/execroot/_main/external/+_repo_rules+clang-linux-x86_64/include/c++/v1/memory

[armandomontanez]

Clang is resolving the e38ec8e746bf3d11672a536413f07932/sandbox/linux-sandbox/113/execroot/_main to the real path e38ec8e746bf3d11672a536413f07932/external/+_repo_rules+clang-linux-x86_64/bin/llvm when it hands off to llvm

[armandomontanez]

Fun finding, you can force include paths to be relative with clang via -resource-dir and -ccc-install-dir. Since I'm interested in resolving this for cases where flags like this aren't respected, though, I'm just going to keep digging.

~/.cache/bazel/_bazel_amontanez/e38ec8e746bf3d11672a536413f07932/sandbox/linux-sandbox/113/execroot/_main$ external/+_repo_rules+clang-linux-x86_64/bin/clang -E -x c++ - -v -resource-dir=external/+_repo_rules+clang-linux-x86_64/lib/clang/19 -ccc-install-dir external/+_repo_rules+clang-linux-x86_64/bin < /dev/null
Fuchsia clang version 19.0.0git (https://llvm.googlesource.com/llvm-project 0cfd03ac0d3f9713090a581bda07584754c73a49)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: external/+_repo_rules+clang-linux-x86_64/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/13
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/14
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/14
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Candidate multilib: x32;@mx32
Selected multilib: .;@m64
 (in-process)
 "/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/e38ec8e746bf3d11672a536413f07932/external/+_repo_rules+clang-linux-x86_64/bin/llvm" "clang" -cc1 -triple x86_64-unknown-linux-gnu -E -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=all -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -debugger-tuning=gdb -fdebug-compilation-dir=/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/e38ec8e746bf3d11672a536413f07932/sandbox/linux-sandbox/113/execroot/_main -v -fcoverage-compilation-dir=/usr/local/google/home/amontanez/.cache/bazel/_bazel_amontanez/e38ec8e746bf3d11672a536413f07932/sandbox/linux-sandbox/113/execroot/_main -resource-dir external/+_repo_rules+clang-linux-x86_64/lib/clang/19 -internal-isystem external/+_repo_rules+clang-linux-x86_64/bin/../include/x86_64-unknown-linux-gnu/c++/v1 -internal-isystem external/+_repo_rules+clang-linux-x86_64/bin/../include/c++/v1 -internal-isystem external/+_repo_rules+clang-linux-x86_64/lib/clang/19/include -internal-isystem /usr/local/include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/14/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdeprecated-macro -ferror-limit 19 -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fcxx-exceptions -fexceptions -fcolor-diagnostics -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o - -x c++ -
clang -cc1 version 19.0.0git based upon LLVM 19.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/14/../../../../x86_64-linux-gnu/include"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 external/+_repo_rules+clang-linux-x86_64/bin/../include/x86_64-unknown-linux-gnu/c++/v1
 external/+_repo_rules+clang-linux-x86_64/bin/../include/c++/v1
 external/+_repo_rules+clang-linux-x86_64/lib/clang/19/include
 /usr/local/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
# 1 "<stdin>"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 463 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "<stdin>" 2

[rickeylev]

Would this issue also explain an error like this?

the source file '<various>' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain):
  '/usr/lib/gcc/x86_64-linux-gnu/14/include/stddef.h'
  '/usr/lib/gcc/x86_64-linux-gnu/14/include/limits.h'
  '/usr/lib/gcc/x86_64-linux-gnu/14/include/syslimits.h'
  '/usr/lib/gcc/x86_64-linux-gnu/14/include/stdarg.h'

While trying to get stardoc working with Bazel 8, this error occurs when building protobuf, absl, or one of its dependencies. The <various> file changes every few invocations, which makes me think its a lower-level problem in the rules, rather than a particular library.

[armandomontanez]

Would this issue also explain an error like this?

the source file '<various>' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain):
  '/usr/lib/gcc/x86_64-linux-gnu/14/include/stddef.h'
  '/usr/lib/gcc/x86_64-linux-gnu/14/include/limits.h'
  '/usr/lib/gcc/x86_64-linux-gnu/14/include/syslimits.h'
  '/usr/lib/gcc/x86_64-linux-gnu/14/include/stdarg.h'

While trying to get stardoc working with Bazel 8, this error occurs when building protobuf, absl, or one of its dependencies. The <various> file changes every few invocations, which makes me think its a lower-level problem in the rules, rather than a particular library.

That appears to be unrelated

[armandomontanez]

@finn-ball started working on a gcc example using the gcc toolchains from https://toolchains.bootlin.com/, which appear to fall under this category of "toolchains that never respect -no-canonical-prefixes and need to be able to allowlist_include_directories"

It looks like allowlist_include_directories does not work with that as well, though possibly for slightly different reasons (the paths didn't appear to ever be resolved, and had many ../ in them

[armandomontanez]

We were able to work around the issue with the bootlin toolchains by directly calling ./bazel-rule_based_toolchain/external/+_repo_rules+gcc-linux-x86_64/bin/x86_64-linux-gcc.br_real, which is what gcc was wrapping and calling out to.

[armandomontanez]

I think the solution is to find some way to get the realpath of any allowlisted builtin include directories before adding them to cxx_builtin_include_directories. I'm also curious to try the %package(@toolchain_distribution//)%/includes somehow, which was previously used but abandoned after switching to directory rules from skylib.