crash in Aws::Utils::Threading::PooledThreadExecutor #1776
Comments
fboranek
added
bug
|
Thanks for the detailed write up and PR. I'll see if we can get it reviewed and merged. |
jmklix
added
pending-release
|
Hello, is there an update on this bug fix? I am seeing the same issue. Thanks! |
|
Hi, Please call pTransferManager->WaitUntilAllFinished() once you've finished the file transfer, or before leaving the scope of It does not fix the original issue reported here though, we will also check the original/root issue of PooledThreadExecutor cyclic references. |
|
Noted, thank you for the update |
I think, it cannot be solved otherwise than by suggest MR. The main issue is passing pool as shared pointer. Then it cannot be guaranteed lifetime of object in main thread. int main() {
struct Pool {
std::thread thread;
~Pool()
{
if (thread.joinable()) thread.join();
std::cout << "~Pool() finished\n";
}
};
std::cout << "1. simple\n";
{
auto pool = std::make_shared<Pool>();
pool->thread = std::thread{[=]() {
std::this_thread::sleep_for(std::chrono::seconds{5});
std::cout << "Task finished\n";
}};
}
std::cout << "2. with copy\n";
{
auto pool = std::make_shared<Pool>();
pool->thread = std::thread{[=]() {
auto copy = pool;
std::this_thread::sleep_for(std::chrono::seconds{5});
std::cout << "Task finished\n";
}};
}
std::cout << std::flush;
}with output The second thread was still running when the process ended because i copied shared pointer into task. |
|
This should be fixed with this merged PR: #2787 |
|
This issue is now closed. Comments on closed issues are hard for our team to see. |
Confirm by changing [ ] to [x] below to ensure that it's a bug:
Describe the bug
When the application destroy shared pointer to
PooledThreadExecutorand the executor has still some running tasks it will lead to crash.SDK version number
1.8.185
Platform/OS/Hardware/Device
Linux / Debian Buster
To Reproduce (observed behavior)
Consider following code
{ { std::shared_ptr<Aws::Utils::Threading::PooledThreadExecutor> executor; // initialize executor std::shared_ptr<Aws::S3::S3Client> s3Client; // initialize client Aws::Transfer::TransferManagerConfiguration transfer_config(executor.get()); transfer_config.s3Client = s3Client; // just to reproduce the problem which rarely happen transfer_config.transferStatusUpdatedCallback = [&](const Aws::Transfer::TransferManager*, const std::shared_ptr<const Aws::Transfer::TransferHandle>& handle) { if (handle->GetStatus() == Aws::Transfer::TransferStatus::COMPLETED) std::this_thread::sleep_for(std::chrono::seconds(30)); }; auto transferManager = Aws::Transfer::TransferManager::Create(transfer_config); auto handle = transferManager->UploadFile(/* args */); handle->WaitUntilFinished(); } std::this_thread::sleep_for(std::chrono::seconds(60)); }which can lead to the appended crash.
The main problem is
WaitUntilFinished()is notified from lambda in the executor. Particularly fromTransferManager::Handle PutObjectResponse, so it stops blocking. All shared pointer are destroyed, but it doesn't mean that also Instances are destroyed. And here is a race condition inPooledThreadExecutor.Aws::Delete(fn);then refcount of shared pointerPooledThreadExecutoris 1 and it will be destroyed in main thread - this is correct behaviorPooledThreadExecutoris higher then 1 and it keeps living in a lamda. Later, deleting the lambda triggers also destroying the executor. Destructor of executor blocks calling thread, because it join all his threads. But her the calling thread is from task and tries to join him self. Yes, in coredump the thread number is the same. The exception rised fromstd::thread::joinisstd::system_errorpthread_join error : Resource deadlock avoidedExpected behavior
I don't see any fix except a public function in
PooledThreadExecutorwhich will block until the executor is fully drained. E.g.{ { std::shared_ptr<Aws::Utils::Threading::PooledThreadExecutor> executor; // initialize executor ... handle->WaitUntilFinished(); executor->WaitUntilStopped(); } std::this_thread::sleep_for(std::chrono::seconds(60)); }or any other method to safely destroy the executor.
Additional context
backtrace from coredump
The text was updated successfully, but these errors were encountered: