Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(thirdparty): Use facebook/rocksdb and pegasus-kv/encfs as rocksdb thirdparty #1678

Merged
merged 4 commits into from
Nov 9, 2023
Merged

feat(thirdparty): Use facebook/rocksdb and pegasus-kv/encfs as rocksdb thirdparty #1678

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
cf06009
init
acelyc111 Nov 8, 2023
b29f833
feat: Use facebook/rocksdb and pegasus-kv/encfs as rocksdb thirdparty
acelyc111 Nov 8, 2023
e1d473f
disable macOS
acelyc111 Nov 9, 2023
c65ad47
adapt enc uri
acelyc111 Nov 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions src/utils/env.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ DSN_DEFINE_bool(pegasus.server,

DSN_DEFINE_string(pegasus.server,
server_key_for_testing,
"server_key_for_testing",
"0123456789ABCDEF0123456789ABCDEF",
"The encrypted server key to use in the filesystem. NOTE: only for testing.");

DSN_DEFINE_string(pegasus.server,
Expand All @@ -62,8 +62,9 @@ rocksdb::Env *NewEncryptedEnv()
{
// Create an encryption provider.
std::shared_ptr<rocksdb::EncryptionProvider> provider;
auto provider_id =
fmt::format("AES:{},{}", FLAGS_server_key_for_testing, FLAGS_encryption_method);
auto provider_id = fmt::format("id=AES;hex_instance_key={};method={}",
FLAGS_server_key_for_testing,
FLAGS_encryption_method);
auto s = rocksdb::EncryptionProvider::CreateFromString(
rocksdb::ConfigOptions(), provider_id, &provider);
CHECK(s.ok(), "Failed to create encryption provider: {}", s.ToString());
Expand Down
31 changes: 19 additions & 12 deletions thirdparty/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -372,15 +372,7 @@ ExternalProject_Add(jemalloc

option(ROCKSDB_PORTABLE "Minimum CPU arch to support, or 0 = current CPU, 1 = baseline CPU" 0)

# This is the commit "Update the status badge to pegasus-kv/rocksdb's own (#18) " on
# branch v8.5.3-pegasus-encrypt of https://github.com/pegasus-kv/rocksdb.git.
# The v8.5.3-pegasus-encrypt branch is based on the v8.5.3 tag of facebook/rocksdb and add
# the encryption feature.
ExternalProject_Add(rocksdb
URL https://github.com/pegasus-kv/rocksdb/archive/e72106597e6b7924485cadc2433b66029f1aaf17.zip
URL_MD5 6f6daef703586ce788643bbb8a984fff
DEPENDS jemalloc
CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${TP_OUTPUT}
set(ROCKSDB_OPTIONS
-DFAIL_ON_WARNINGS=OFF
-DWITH_BENCHMARK_TOOLS=OFF
-DWITH_TOOLS=OFF
Expand All @@ -391,14 +383,29 @@ ExternalProject_Add(rocksdb
-DWITH_TESTS=OFF
-DWITH_GFLAGS=OFF
-DUSE_RTTI=ON
-DWITH_OPENSSL=ON # enable encryption
-DCMAKE_BUILD_TYPE=Release
-DWITH_JEMALLOC=${USE_JEMALLOC}
-DJEMALLOC_ROOT_DIR=${TP_OUTPUT}
-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
-DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
-DPORTABLE=${ROCKSDB_PORTABLE}
)
-DPORTABLE=${ROCKSDB_PORTABLE})
# Link error on MacOS, disable building encfs plugin.
# See https://github.com/pegasus-kv/encfs/issues/4
if (NOT APPLE)
set(ROCKSDB_OPTIONS
${ROCKSDB_OPTIONS}
-DROCKSDB_PLUGINS=encfs)
endif ()
ExternalProject_Add(rocksdb
URL ${OSS_URL_PREFIX}/rocksdb-v8.5.3.tar.gz
https://github.com/facebook/rocksdb/archive/refs/tags/v8.5.3.tar.gz
URL_MD5 f03eac50ec958a21a7cb56183afb8fe4
PATCH_COMMAND ""
COMMAND rm -rf ${TP_DIR}/build/Source/rocksdb/plugin/encfs
COMMAND git clone -b main --depth=1 https://github.com/pegasus-kv/encfs.git ${TP_DIR}/build/Source/rocksdb/plugin/encfs
DEPENDS googletest jemalloc
CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${TP_OUTPUT}
${ROCKSDB_OPTIONS})

ExternalProject_Add(http-parser
URL ${OSS_URL_PREFIX}/http-parser-2.9.4.zip
Expand Down