IO issues on macOS arm64 VMs

[meteorcloudy]

There are many IO related flaky errors on arm64 Mac machines, for examples:

• https://buildkite.com/bazel/google-bazel-presubmit/builds/87931#01942b88-73ce-4bbc-ab0e-e33f42e6032f

ld: file cannot be open()ed, errno=5 path=/Applications/Xcode15.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/15.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib in '/Applications/Xcode15.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/15.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib'
clang: error: linker command failed with exit code 1 (use -v to see invocation)

• https://buildkite.com/bazel/bazel-bazel-examples/builds/3966#01942b82-e594-454c-baac-68ab0b847f52

(01:01:32) ERROR: /Users/buildkite/builds/bk-macos-arm64-ul53/bazel/bazel-bazel-examples/android/ndk/app/src/main/BUILD.bazel:22:15: Linking _nativedeps/669710a0.so failed: error reading file '@@androidndk//toolchains/llvm/prebuilt/darwin-x86_64/sysroot:usr/lib/i686-linux-android/libandroid_support.a': Input/output error
(01:01:32) ERROR: /Users/buildkite/builds/bk-macos-arm64-ul53/bazel/bazel-bazel-examples/android/ndk/app/src/main/BUILD.bazel:22:15: Linking _nativedeps/669710a0.so failed: 1 input file(s) are in error

https://storage.googleapis.com/bazel-untrusted-buildkite-artifacts/019440ca-e06d-4e00-a88e-7caf087930a2/src/test/shell/bazel/bazel_determinism_test/test_attempts/attempt_1.log

find: sha1sum: Input/output error
find: sha1sum: Input/output error
find: sha1sum: Input/output error
find: sha1sum: Input/output error
find: sha1sum: Input/output error

[meteorcloudy]

@fweikert

[linzhp]

We are seeing similar macOS CI issue on rules_go too. It can be reproduced after disabling remote cache: bazel-contrib/rules_go#4216

[peng3141]

@meteorcloudy @fweikert could you share a status on this? thanks.

[meteorcloudy]

I'm still looking into this, definitely some infra problem.