-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Aws::Crt::Io::ClientBootstrap destructor may launch thread at process exit, and crash #1809
Comments
Does not happen with SDK version 1.8.186. |
In case it could help narrow down the source of the bug, I tested a few different versions of aws-sdk-cpp on CentOS 7:
|
Hi @jdblischak , |
The fix on the conda-forge side was to revert back to the 1.8.186 SDK version. With the current issue, we cannot use a newer SDK on Windows. |
@KaibaLopez Thanks for following up
As @xhochy commented, the conda-forge workaround is to pin to an older version of aws-sdk-cpp. Personally I fixed it by specifying
I was able to reproduce the bug using the code below:
where
Here is the full error and traceback that I observe:
|
Update: this error has been reported on Ubuntu and CentOS as well: https://issues.apache.org/jira/browse/ARROW-15141 |
@ihnorton have you run into something similar for tiledb? |
No, we are still on 1.8, and that backtrace does not ring a bell. |
Hi. Just to say we have hit this exact same issue using v1.9.72 within our in-house build at MathWorks. |
We also see this on windows now (while updating).
|
Because the latest AWS SDK C++ has a problem: aws/aws-sdk-cpp#1809
Because the latest AWS SDK C++ has a problem: aws/aws-sdk-cpp#1809 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Was this resolved by awslabs/aws-c-io#515? |
Because the latest AWS SDK C++ has a problem: aws/aws-sdk-cpp#1809 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
I don't know this repo, but from looking around, it seems that:
The last update of the aws-c-io version used in this repo was in June (NB: at which point 5 newer releases of aws-c-io would have been already available). Perhaps @sdavtaker (author of last update) can illuminate the process of what's necessary to update the respective dependencies. This issue is biting us quite hard in conda-forge, made worse by the fact that aws-sdk-cpp 1.8 does not seem compatible anymore with current versions of the rest of the aws-c-* stack (which we need to unbundle for several reasons). I also noticed still regular discussions about this problem in other repos, e.g. huggingface/datasets#3310
So it would be really good if we could upgrade aws-c-io here and then determine if that actually fixes things... |
pyarrow 10.0.1 was just released in conda-forge, which is the first release where we're building against aws-sdk-cpp 1.9.* again after more than a year. Since we cannot test the failure reported here on our infra, I'd be very grateful if someone could verify that the problem does or doesn't reappear. 🙃
Edit: if things are fine, I'm happy to backport this to arrow 6.x-9.x. |
Confirmed. Thanks @h-vetinari! See reproducible example at conda-forge/arrow-cpp-feedstock#567 (comment) |
In case someone else is still facing it... I had the same issue, but it was caused because https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/basic-use.html |
As @cardinotGV said please make sure that you are calling #include <aws/core/Aws.h>
int main(int argc, char** argv)
{
Aws::SDKOptions options;
Aws::InitAPI(options);
{
// make your SDK calls here.
}
Aws::ShutdownAPI(options);
return 0;
} If you are still running into any crashes at process exit please let me know |
|
Confirm by changing [ ] to [x] below to ensure that it's a bug:
Describe the bug
We're getting a user report of a crash, seemingly at process shutdown, on Windows:
conda-forge/arrow-cpp-feedstock#567
Apparently the
ClientBootstrap
destructor can indirectly trigger the launch of a new thread usingaws_thread_launch
. The thread launch fails at process shutdown, at least on Windows, triggering an assertion error and therefore a process crash.SDK version number
1.9.120
Platform/OS/Hardware/Device
Windows/10.0.17763
(also reported on CentOS 8 and Ubuntu: https://issues.apache.org/jira/browse/ARROW-15141)
To Reproduce (observed behavior)
Basically conda-forge/arrow-cpp-feedstock#567 (comment), but I'm not sure what the exact steps are (I'm not the original reporter).
Expected behavior
Failing to launch a thread at process shutdown should probably not crash the process.
The text was updated successfully, but these errors were encountered: