Multiple versions of AWS-SDK?

[jonkeane]

Thanks for the heads up about bumping the AWS SDK version in #94. Arrow depends on a slightly older version of the sdk (1.8.133) and when trying to use higher versions ran into a bug that is not yet resolved in the AWS sdk

It looks like the newer version and the older version are different enough that we're getting a symbol look up error starting when #94 merged

** testing if installed package can be loaded from temporary location
Error: Error: package or namespace load failed for ‘arrow’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/Users/runner/work/crossbow/crossbow/arrow/r/check/arrow.Rcheck/00LOCK-arrow/00new/arrow/libs/arrow.so':
  dlopen(/Users/runner/work/crossbow/crossbow/arrow/r/check/arrow.Rcheck/00LOCK-arrow/00new/arrow/libs/arrow.so, 6): Symbol not found: __ZN3Aws3Crt11g_allocatorE
  Referenced from: /Users/runner/work/crossbow/crossbow/arrow/r/check/arrow.Rcheck/00LOCK-arrow/00new/arrow/libs/arrow.so
  Expected in: flat namespace
 in /Users/runner/work/crossbow/crossbow/arrow/r/check/arrow.Rcheck/00LOCK-arrow/00new/arrow/libs/arrow.so
Error: Error: loading failed

Would it be possible or advisable to have a bottle of an older version of aws-sdk-cpp available that we could pin the apache-arrow formula to?

[jeroen]

We can just revert it for now if that helps you. I'm not sure how you manage to get that linking error though, are you compiling against vendored aws-sdk headers and then linking against the autobrew library? I didn't get any linking errors trying to rebuild arrow (in autobrew) against the new aws-sdk-cpp (in autobrew).

[jonkeane]

I'm still trying to work out how we're getting the symbol error as well (and trying to reproduce this locally too, though running into unrelated issues so far).

What I have observed so far:

The symbol __ZN3Aws3Crt11g_allocatorE looks like it's coming from aws-crt-cpp (which is a new sub-component of aws-sdk-cpp starting in 1.9), but strangely I'm not seeing that in the list of linked libraries during the build:

clang++ -mmacosx-version-min=10.13 -std=gnu++11 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o arrow.so RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o bridge.o buffer.o chunkedarray.o compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o expression.o feather.o field.o filesystem.o imports.o io.o json.o memorypool.o message.o parquet.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T//build-apache-arrow/lib -larrow_dataset -lparquet -lbrewparquet -lbrewarrow_dataset -lbrewarrow -lbrewarrow_bundled_dependencies -lbrewthrift -lbrewlz4 -lbrewsnappy -lbrewzstd -lbrewaws-cpp-sdk-config -lbrewaws-cpp-sdk-transfer -lbrewaws-cpp-sdk-identity-management -lbrewaws-cpp-sdk-cognito-identity -lbrewaws-cpp-sdk-sts -lbrewaws-cpp-sdk-s3 -lbrewaws-cpp-sdk-core -lbrewaws-c-event-stream -lbrewaws-checksums -lbrewaws-c-common -lpthread -lcurl -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation

I've tried adding it to our link libs (though this won't work for pre 1.8 aws-sdk versions anyway, it still didn't succeed at linking / finding that correctly: https://github.com/ursacomputing/crossbow/runs/4708595515?check_suite_focus=true#step:7:4059

Also, the error pops up when trying to load the package after it's installed, and maybe the test step of the apache-arrow formula we have doesn't trigger this symbol load issue the same way that loading via R does?

[jeroen]

I think your CI hardcodes the macos linker flags here: https://github.com/apache/arrow/blob/1440d5ab95c423003ecbe06372663a85cfccf769/r/tools/autobrew#L50
So you probably need to add -laws-crt-cpp there. For static linking the order matters, so I think you need to add it after the other aws libs.

For the deployed version we set the same flags here: https://github.com/autobrew/scripts/blob/master/apache-arrow

[jonkeane]

Aaah, thanks for that!

I did add them, and we do get further (or at least a different symbol being not found):

dlopen(/Users/runner/work/crossbow/crossbow/arrow/r/check/arrow.Rcheck/00LOCK-arrow/00new/arrow/libs/arrow.so, 6): Symbol not found: _aws_default_dns_resolve

I'm still searching to figure out if there's a different module that adding would add that...

[jonkeane]

Ah ok, finally got them (I ended up searching the bottle for symbols along the way, though it looks like we needed one for each of the submodules in aws-crt-cpp...)

Adding -laws-crt-cpp -laws-c-io -laws-c-s3 -laws-c-auth -laws-c-http -laws-c-cal -laws-c-compression -laws-c-mqtt appears to get it working: https://github.com/ursacomputing/crossbow/runs/4720019596?check_suite_focus=true

I'm still not sure why the build of apache-arrow doesn't have this issue when it is tested.

I need to dig a bit more to see if the bug reported in AWS-SDK 1.9 might impact folks on macOS (or if it's just linux OSes) to figure out if we should roll back to a lower version. But at least this seems to work otherwise.

[jeroen]

OK thanks.

If the issue may not exist on macos/clang, I would prefer to not roll back such that we stay closer in sync with upstream homebrew, which currently has aws-sdk-cpp 1.9.150. That is the version that the homebrew build of apache-arrow is using as well so if there was a bug we would probably have heard about it?

If we do revert, we may also need to roll back autobrew/homebrew-cran#39

[jonkeane]

I chatted with @pitrou about this and he reminded me we've seen the AWS SDK bug on both Windows and CentOS. It's possible it is OS specific, but we aren't sure it won't crop up on macOS.

If you would prefer, I'm fine leaving this for now and if we run into any issues going forward, downgrading it then. We are spinning up to have an Arrow release, so we will have more checks running generally (and will have a CRAN submission after). Thoughts?

[jeroen]

Yes let's leave it for now, hopefully there is no issue on macos/clang, or maybe the r bindings are not affected by the threading issue in the first place.

If we do see the issue we can downgrade autobrew later on, though of course we won't be able to help folks that compile arrow from source against homebrew, which include aws-sdk-cpp 1.9.