Refactor ignoreCorruptFiles #10
Conversation
|
|
XinShuoWang
force-pushed
the
refactor_ignore_corrupt_file
branch
from
6ac583f to
6831f2a
Compare
XinShuoWang
changed the title
XinShuoWang
force-pushed
the
refactor_ignore_corrupt_file
branch
from
6831f2a to
68bbbad
Compare
XinShuoWang
force-pushed
the
refactor_ignore_corrupt_file
branch
from
68bbbad to
ffed9ba
Compare
XinShuoWang
force-pushed
the
refactor_ignore_corrupt_file
branch
from
dc0a970 to
56092cf
Compare
| // preload can fail in async thread, then retry in sync load | ||
| if (isAsyncPreloadThread() && | ||
| std::string(e.what()).find(kAsyncPreloadThreadName) != | ||
| std::string::npos) { |
There was a problem hiding this comment.
Why std::string(e.what()).find(kAsyncPreloadThreadName) != std::string::npos was removed here?
There was a problem hiding this comment.
Why
std::string(e.what()).find(kAsyncPreloadThreadName) != std::string::nposwas removed here?
Ensure that Scan-related errors can be propagated upwards to bolt/connectors/hive/HiveDataSource.cpp and handled, so that the IgnoreCorruptFile-related logic doesn't need to be added in the asynchronous Preload logic.
| const ::testing::TestInfo* test_info = | ||
| ::testing::UnitTest::GetInstance()->current_test_info(); |
There was a problem hiding this comment.
Do we have gtest dependency in a build without test? Can we wrapper test related logic inside a macro? So that we can avoid gtest dependency in release build.
There was a problem hiding this comment.
Do we have gtest dependency in a build without test? Can we wrapper test related logic inside a macro? So that we can avoid gtest dependency in release build.
- I didn't change the link settings when compile release version, so I think
gtestwas already linked by default in the original configuration. - I think it's great that you pointed this out, and I will try to remove the gtest dependency here.
There was a problem hiding this comment.
Do we have gtest dependency in a build without test? Can we wrapper test related logic inside a macro? So that we can avoid gtest dependency in release build.
I have completed the modifications in this commit:7d4c8b1
4 participants
In Spark, there is an feature named ignoreCorruptFile, which allows you to using the configuration
spark.sql.files.ignoreCorruptFilesto ignore corrupt files while reading data from files.Bolt has implemented this feature, but the code is not clean. I want to refactor this feature, including below points: