Broken linkage that affects gdal and apache-arrow

[hxmn]

brew gist-logs <formula> link OR brew config AND brew doctor output

brew config:


❯ brew config
HOMEBREW_VERSION: 4.1.6-2-g8dd2073
ORIGIN: https://github.com/Homebrew/brew
HEAD: 8dd2073d84b07133dcf3e6e25709eb280826f769
Last commit: 4 hours ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 6d7e6da763d9c01a8da5edc176f1821e187409ba
Core tap last commit: 23 minutes ago
Core tap branch: master
Core tap JSON: 21 Aug 18:03 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_BAT_THEME: 1337
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: /private/tmp/com.apple.launchd.EEngx7o9K0/org.xquartz:0
HOMEBREW_MAKE_JOBS: 10
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: 10-core 64-bit arm_firestorm_icestorm
Clang: 14.0.3 build 1403
Git: 2.39.2 => /Library/Developer/CommandLineTools/usr/bin/git
Curl: 7.88.1 => /usr/bin/curl
macOS: 13.4-arm64
CLT: 14.3.1.0.1.1683849156
Xcode: 14.3.1
Rosetta 2: false

brew doctor:

Your system is ready to brew.

Verification

• My "brew doctor output" says Your system is ready to brew. and am still able to reproduce my issue.
• I ran brew update and am still able to reproduce my issue.
• I have resolved all warnings from brew doctor and that did not fix my problem.
• I searched for recent similar issues at https://github.com/Homebrew/homebrew-core/issues?q=is%3Aissue and found no duplicates.

What were you trying to do (and why)?

I've upgraded from gdal version 3.6.4 to 3.7.1_1 and now it's broken my python script that uses gdal. I've upgraded because forgot to disable auto updater :(

What happened (include all command output)?

When I run my python script and I see next stacktrace:

...
  File "/Users/username/.pyenv/versions/3.10.8/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/opt/homebrew/opt/gdal/lib/libgdal.dylib, 0x0006): Symbol not found: __ZN3Aws6Client19ClientConfigurationC1Ev
  Referenced from: <FA8C3295-2793-3C69-A419-16C41753696B> /opt/homebrew/Cellar/apache-arrow/12.0.1_4/lib/libarrow.1200.1.0.dylib
  Expected in:     <BDB1F1E3-0BE9-3D7D-A57E-9D9F8CAD197A> /opt/homebrew/Cellar/aws-sdk-cpp/1.11.145/lib/libaws-cpp-sdk-core.dylib

What did you expect to happen?

Previous version of gdal (3.6.4) has been without this issue.

Step-by-step reproduction instructions (by running brew commands)

Despite that I've faced this issue on my Mac M1, it can be reproduced on linux in docker environment. Here is the steps:

1. Run shell inside last docker image: docker run -it homebrew/brew:4.1.6 /bin/bash
2. Install gdal: brew install gdal (it takes about 10 minutes)
3. Try to load gdal library in python with this oneliner:

$(brew --prefix python)/bin/python3 -c "from ctypes import CDLL;CDLL('`brew --prefix gdal`/lib/libgdal.so')"

The output:

linuxbrew@43d6ad3e6252:~$ $(brew --prefix python)/bin/python3 -c "from ctypes import CDLL;CDLL('`brew --prefix gdal`/lib/libgdal.so')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/linuxbrew/.linuxbrew/opt/python@3.11/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /home/linuxbrew/.linuxbrew/opt/apache-arrow/lib/libarrow.so.1200: undefined symbol: _ZN3Aws6Client19ClientConfigurationC1Ev

To reproduce it on Mac use this steps:

1. Install gdal: brew install gdal
2. Try to load gdal library in python

from ctypes import CDLL
CDLL('/opt/homebrew/opt/gdal/lib/libgdal.dylib')

Or use this oneliner: python -c "from ctypes import CDLL;CDLL('/opt/homebrew/opt/gdal/lib/libgdal.dylib')"

It results in:

>>> CDLL('/opt/homebrew/opt/gdal/lib/libgdal.dylib')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/username/.pyenv/versions/3.10.8/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/opt/homebrew/opt/gdal/lib/libgdal.dylib, 0x0006): Symbol not found: __ZN3Aws6Client19ClientConfigurationC1Ev
  Referenced from: <FA8C3295-2793-3C69-A419-16C41753696B> /opt/homebrew/Cellar/apache-arrow/12.0.1_4/lib/libarrow.1200.1.0.dylib
  Expected in:     <BDB1F1E3-0BE9-3D7D-A57E-9D9F8CAD197A> /opt/homebrew/Cellar/aws-sdk-cpp/1.11.145/lib/libaws-cpp-sdk-core.dylib

[ShotSkydiver]

Same thing happening here! Trying to downgrade GDAL to no avail.

[ShotSkydiver]

I was able to fix this by downgrading GDAL to 3.6.4, libspatialite to 5.0.1, and aws-sdk-cpp to 1.11.140, finally. Uninstalled all three and downloaded the earlier versions of the .rb files for each formula from the Homebrew repo and installed via those files.

[nattyg93]

@p-linnane (tagging you as the author of the commit updating aws-sdk-cpp to 1.11.145)

This issue should be resolved in version 1.11.147 as per aws/aws-sdk-cpp#2632.

Although I did notice this comment in aws-sdk-cpp.rb:

homebrew-core/Formula/a/aws-sdk-cpp.rb

Line 4 in 897b361

# aws-sdk-cpp should only be updated every 5 releases on multiples of 5

I'm hoping that an exception is made for 1.11.147.

[p-linnane]

Thanks for the ping. Taking a look at this now. We can make adjustments to get the fixed version shipped once it's released upstream.

[ZhongRuoyu]

This should be fixed by #140127. Please do brew update && brew upgrade.