Merge branch 'master' into rocksdict

Author: Congyuwang

.github/workflows/rust.yml

@@ -2,7 +2,7 @@ name: RocksDB CI
 
 on: [push, pull_request]
 env:
-  RUST_VERSION: 1.60.0
+  RUST_VERSION: 1.63.0
 
 jobs:
   fmt:
@@ -104,6 +104,8 @@ jobs:
         run: |
           cargo test --all
           cargo test --all --features multi-threaded-cf
+      - name: Free disk space
+        run: cargo clean
       - name: Run rocksdb tests (jemalloc)
         if: runner.os != 'Windows'
         run: cargo test --all --features jemalloc

CHANGELOG.md

@@ -2,6 +2,10 @@
 
 ## [Unreleased]
 
+* Bump MSRV to 1.63.0 (mina86)
+* Convert properties to `&PropName` which can be converted at no cost to `&CStr`
+  and `&str` (mina86)
+
 ## 0.21.0 (2023-05-09)
 
 * Add doc-check to CI with fix warnings in docs (YuraKotov)
@@ -109,10 +113,10 @@
 * Bump `librocksdb-sys` up to 6.20.3 (olegnn, akrylysov)
 * Add `DB::key_may_exist_cf_opt` method (stanislav-tkach)
 * Add `Options::set_zstd_max_train_bytes` method (stanislav-tkach)
-* Mark Cache and Env as Send and Sync (akrylysov)  
+* Mark Cache and Env as Send and Sync (akrylysov)
 * Allow cloning the Cache and Env (duarten)
-* Make SSE inclusion conditional for target features (mbargull)  
-* Use Self where possible (adamnemecek)  
+* Make SSE inclusion conditional for target features (mbargull)
+* Use Self where possible (adamnemecek)
 * Don't leak dropped column families (ryoqun)
 
 ## 0.16.0 (2021-04-18)
@@ -169,23 +173,23 @@
 * Add `set_max_total_wal_size` to the `Options` (wqfish)
 * Simplify conversion on iterator item (zhangsoledad)
 * Add `flush_cf` method to the `DB` (wqfish)
-* Fix potential segfault when calling `next` on the `DBIterator` that is at the end of the range (wqfish) 
+* Fix potential segfault when calling `next` on the `DBIterator` that is at the end of the range (wqfish)
 * Move to Rust 2018 (wqfish)
 * Fix doc for `WriteBatch::delete` (wqfish)
 * Bump `uuid` and `bindgen` dependencies (jonhoo)
 * Change APIs that never return error to not return `Result` (wqfish)
 * Fix lifetime parameter for iterators (wqfish)
-* Add a doc for `optimize_level_style_compaction` method (NikVolf)  
+* Add a doc for `optimize_level_style_compaction` method (NikVolf)
 * Make `DBPath` use `tempfile` (jder)
 * Refactor `db.rs` and `lib.rs` into smaller pieces (jder)
 * Check if we're on a big endian system and act upon it (knarz)
 * Bump internal snappy version up to 1.1.8 (aleksuss)
 * Bump rocksdb version up to 6.7.3 (aleksuss)
-* Atomic flush option (mappum)  
+* Atomic flush option (mappum)
 * Make `set_iterate_upper_bound` method safe (wqfish)
 * Add support for data block hash index (dvdplm)
 * Add some extra config options (casualjim)
-* Add support for range delete APIs (wqfish)  
+* Add support for range delete APIs (wqfish)
 * Improve building `librocksdb-sys` with system libraries (basvandijk)
 * Add support for `open_for_read_only` APIs (wqfish)
 * Fix doc for `DBRawIterator::prev` and `next` methods (wqfish)

Cargo.toml

@@ -3,7 +3,7 @@ name = "rocksdb"
 description = "Rust wrapper for Facebook's RocksDB embeddable database"
 version = "0.21.0"
 edition = "2018"
-rust-version = "1.60"
+rust-version = "1.63"
 authors = ["Tyler Neely <t@jujit.su>", "David Greenberg <dsg123456789@gmail.com>"]
 repository = "https://github.com/rust-rocksdb/rust-rocksdb"
 license = "Apache-2.0"
@@ -36,7 +36,7 @@ serde1 = ["serde"]
 
 [dependencies]
 libc = "0.2"
-librocksdb-sys = { path = "librocksdb-sys", version = "0.11.0" }
+librocksdb-sys = { path = "librocksdb-sys", version = "0.12.0" }
 serde = { version = "1", features = [ "derive" ], optional = true }
 
 [dev-dependencies]

README.md

@@ -5,7 +5,7 @@ rust-rocksdb
 [![documentation](https://docs.rs/rocksdb/badge.svg)](https://docs.rs/rocksdb)
 [![license](https://img.shields.io/crates/l/rocksdb.svg)](https://github.com/rust-rocksdb/rust-rocksdb/blob/master/LICENSE)
 [![Gitter chat](https://badges.gitter.im/rust-rocksdb/gitter.png)](https://gitter.im/rust-rocksdb/lobby)
-![rust 1.60.0 required](https://img.shields.io/badge/rust-1.60.0-blue.svg?label=MSRV)
+![rust 1.63.0 required](https://img.shields.io/badge/rust-1.63.0-blue.svg?label=MSRV)
 
 
 ![GitHub commits (since latest release)](https://img.shields.io/github/commits-since/rust-rocksdb/rust-rocksdb/latest.svg)
@@ -16,25 +16,25 @@ rust-rocksdb
 
 ## Contributing
 
-Feedback and pull requests welcome!  If a particular feature of RocksDB is 
-important to you, please let me know by opening an issue, and I'll 
+Feedback and pull requests welcome!  If a particular feature of RocksDB is
+important to you, please let me know by opening an issue, and I'll
 prioritize it.
 
 ## Usage
 
-This binding is statically linked with a specific version of RocksDB. If you 
-want to build it yourself, make sure you've also cloned the RocksDB and 
+This binding is statically linked with a specific version of RocksDB. If you
+want to build it yourself, make sure you've also cloned the RocksDB and
 compression submodules:
 
     git submodule update --init --recursive
 
 ## Compression Support
-By default, support for the [Snappy](https://github.com/google/snappy), 
-[LZ4](https://github.com/lz4/lz4), [Zstd](https://github.com/facebook/zstd), 
-[Zlib](https://zlib.net), and [Bzip2](http://www.bzip.org) compression 
-is enabled through crate features.  If support for all of these compression 
-algorithms is not needed, default features can be disabled and specific 
-compression algorithms can be enabled. For example, to enable only LZ4 
+By default, support for the [Snappy](https://github.com/google/snappy),
+[LZ4](https://github.com/lz4/lz4), [Zstd](https://github.com/facebook/zstd),
+[Zlib](https://zlib.net), and [Bzip2](http://www.bzip.org) compression
+is enabled through crate features.  If support for all of these compression
+algorithms is not needed, default features can be disabled and specific
+compression algorithms can be enabled. For example, to enable only LZ4
 compression support, make these changes to your Cargo.toml:
 
 ```

librocksdb-sys/Cargo.toml

@@ -1,8 +1,8 @@
 [package]
 name = "librocksdb-sys"
-version = "0.11.0+8.1.1"
+version = "0.12.0+8.5.3"
 edition = "2018"
-rust-version = "1.60"
+rust-version = "1.63"
 authors = ["Karl Hobley <karlhobley10@gmail.com>", "Arkadiy Paronyan <arkadiy@ethcore.io>"]
 license = "MIT/Apache-2.0/BSD-3-Clause"
 description = "Native bindings to librocksdb"

librocksdb-sys/build.rs

@@ -118,24 +118,19 @@ fn build_rocksdb() {
         }
         if target_features.contains(&"sse4.2") {
             config.flag_if_supported("-msse4.2");
-            config.define("HAVE_SSE42", Some("1"));
         }
         // Pass along additional target features as defined in
         // build_tools/build_detect_platform.
         if target_features.contains(&"avx2") {
             config.flag_if_supported("-mavx2");
-            config.define("HAVE_AVX2", Some("1"));
         }
         if target_features.contains(&"bmi1") {
             config.flag_if_supported("-mbmi");
-            config.define("HAVE_BMI", Some("1"));
         }
         if target_features.contains(&"lzcnt") {
             config.flag_if_supported("-mlzcnt");
-            config.define("HAVE_LZCNT", Some("1"));
         }
         if !target.contains("android") && target_features.contains(&"pclmulqdq") {
-            config.define("HAVE_PCLMUL", Some("1"));
             config.flag_if_supported("-mpclmul");
         }
     }

librocksdb-sys/build_version.cc

@@ -8,17 +8,17 @@
 
 // The build script may replace these values with real values based
 // on whether or not GIT is available and the platform settings
-static const std::string rocksdb_build_git_sha  = "6a436150417120a3f9732d65a2a5c2b8d19b60fc";
-static const std::string rocksdb_build_git_tag = "rocksdb_build_git_tag:v8.1.1";
+static const std::string rocksdb_build_git_sha  = "f32521662acf3352397d438b732144c7813bbbec";
+static const std::string rocksdb_build_git_tag = "rocksdb_build_git_tag:v8.5.3";
 #define HAS_GIT_CHANGES 0
 #if HAS_GIT_CHANGES == 0
 // If HAS_GIT_CHANGES is 0, the GIT date is used.
 // Use the time the branch/tag was last modified
-static const std::string rocksdb_build_date = "rocksdb_build_date:2023-04-06 16:38:52";
+static const std::string rocksdb_build_date = "rocksdb_build_date:2023-09-01 20:58:39";
 #else
 // If HAS_GIT_CHANGES is > 0, the branch/tag has modifications.
 // Use the time the build was created.
-static const std::string rocksdb_build_date = "rocksdb_build_date:2023-04-06 16:38:52";
+static const std::string rocksdb_build_date = "rocksdb_build_date:2023-09-01 20:58:39";
 #endif
 
 std::unordered_map<std::string, ROCKSDB_NAMESPACE::RegistrarFunc> ROCKSDB_NAMESPACE::ObjectRegistry::builtins_ = {};

librocksdb-sys/rocksdb

@@ -249,6 +249,8 @@ util/stderr_logger.cc
 util/string_util.cc
 util/thread_local.cc
 util/threadpool_imp.cc
+util/udt_util.cc
+util/write_batch_util.cc
 util/xxhash.cc
 utilities/agg_merge/agg_merge.cc
 utilities/backup/backup_engine.cc

librocksdb-sys/rocksdb_lib_sources.txt

@@ -1072,7 +1072,7 @@ fn ffi() {
                 rocksdb_slicetransform_create_fixed_prefix(3),
             );
             rocksdb_options_set_hash_skip_list_rep(options, 5000, 4, 4);
-            rocksdb_options_set_plain_table_factory(options, 4, 10, 0.75, 16);
+            rocksdb_options_set_plain_table_factory(options, 4, 10, 0.75, 16, 0, 0, 0, 0);
             rocksdb_options_set_allow_concurrent_memtable_write(options, 0);
 
             db = rocksdb_open(options, dbname, &mut err);

librocksdb-sys/tests/ffi.rs

@@ -921,6 +921,28 @@ impl<T: ThreadMode, D: DBInner> DBCommon<T, D> {
         Ok(())
     }
 
+    /// Flushes multiple column families.
+    ///
+    /// If atomic flush is not enabled, it is equivalent to calling flush_cf multiple times.
+    /// If atomic flush is enabled, it will flush all column families specified in `cfs` up to the latest sequence
+    /// number at the time when flush is requested.
+    pub fn flush_cfs_opt(
+        &self,
+        cfs: &[&impl AsColumnFamilyRef],
+        opts: &FlushOptions,
+    ) -> Result<(), Error> {
+        let mut cfs = cfs.iter().map(|cf| cf.inner()).collect::<Vec<_>>();
+        unsafe {
+            ffi_try!(ffi::rocksdb_flush_cfs(
+                self.inner.inner(),
+                opts.inner,
+                cfs.as_mut_ptr(),
+                cfs.len() as libc::c_int,
+            ));
+        }
+        Ok(())
+    }
+
     /// Flushes database memtables to SST files on the disk for a given column family using default
     /// options.
     pub fn flush_cf(&self, cf: &impl AsColumnFamilyRef) -> Result<(), Error> {
@@ -1157,38 +1179,40 @@ impl<T: ThreadMode, D: DBInner> DBCommon<T, D> {
     /// Return the values associated with the given keys and the specified column family
     /// where internally the read requests are processed in batch if block-based table
     /// SST format is used.  It is a more optimized version of multi_get_cf.
-    pub fn batched_multi_get_cf<K, I>(
+    pub fn batched_multi_get_cf<'a, K, I>(
         &self,
         cf: &impl AsColumnFamilyRef,
         keys: I,
         sorted_input: bool,
     ) -> Vec<Result<Option<DBPinnableSlice>, Error>>
     where
-        K: AsRef<[u8]>,
-        I: IntoIterator<Item = K>,
+        K: AsRef<[u8]> + 'a + ?Sized,
+        I: IntoIterator<Item = &'a K>,
     {
         self.batched_multi_get_cf_opt(cf, keys, sorted_input, &ReadOptions::default())
     }
 
     /// Return the values associated with the given keys and the specified column family
     /// where internally the read requests are processed in batch if block-based table
-    /// SST format is used.  It is a more optimized version of multi_get_cf_opt.
-    pub fn batched_multi_get_cf_opt<K, I>(
+    /// SST format is used. It is a more optimized version of multi_get_cf_opt.
+    pub fn batched_multi_get_cf_opt<'a, K, I>(
         &self,
         cf: &impl AsColumnFamilyRef,
         keys: I,
         sorted_input: bool,
         readopts: &ReadOptions,
     ) -> Vec<Result<Option<DBPinnableSlice>, Error>>
     where
-        K: AsRef<[u8]>,
-        I: IntoIterator<Item = K>,
+        K: AsRef<[u8]> + 'a + ?Sized,
+        I: IntoIterator<Item = &'a K>,
     {
-        let (keys, keys_sizes): (Vec<Box<[u8]>>, Vec<_>) = keys
+        let (ptr_keys, keys_sizes): (Vec<_>, Vec<_>) = keys
             .into_iter()
-            .map(|k| (Box::from(k.as_ref()), k.as_ref().len()))
+            .map(|k| {
+                let k = k.as_ref();
+                (k.as_ptr() as *const c_char, k.len())
+            })
             .unzip();
-        let ptr_keys: Vec<_> = keys.iter().map(|k| k.as_ptr() as *const c_char).collect();
 
         let mut pinned_values = vec![ptr::null_mut(); ptr_keys.len()];
         let mut errors = vec![ptr::null_mut(); ptr_keys.len()];
@@ -1796,7 +1820,7 @@ impl<T: ThreadMode, D: DBInner> DBCommon<T, D> {
             ))),
         };
         unsafe {
-            libc::free(value as *mut c_void);
+            ffi::rocksdb_free(value as *mut c_void);
         }
         result
     }
@@ -1989,6 +2013,47 @@ impl<T: ThreadMode, D: DBInner> DBCommon<T, D> {
         }
     }
 
+    /// Obtains the LSM-tree meta data of the default column family of the DB
+    pub fn get_column_family_metadata(&self) -> ColumnFamilyMetaData {
+        unsafe {
+            let ptr = ffi::rocksdb_get_column_family_metadata(self.inner.inner());
+
+            let metadata = ColumnFamilyMetaData {
+                size: ffi::rocksdb_column_family_metadata_get_size(ptr),
+                name: from_cstr(ffi::rocksdb_column_family_metadata_get_name(ptr)),
+                file_count: ffi::rocksdb_column_family_metadata_get_file_count(ptr),
+            };
+
+            // destroy
+            ffi::rocksdb_column_family_metadata_destroy(ptr);
+
+            // return
+            metadata
+        }
+    }
+
+    /// Obtains the LSM-tree meta data of the specified column family of the DB
+    pub fn get_column_family_metadata_cf(
+        &self,
+        cf: &impl AsColumnFamilyRef,
+    ) -> ColumnFamilyMetaData {
+        unsafe {
+            let ptr = ffi::rocksdb_get_column_family_metadata_cf(self.inner.inner(), cf.inner());
+
+            let metadata = ColumnFamilyMetaData {
+                size: ffi::rocksdb_column_family_metadata_get_size(ptr),
+                name: from_cstr(ffi::rocksdb_column_family_metadata_get_name(ptr)),
+                file_count: ffi::rocksdb_column_family_metadata_get_file_count(ptr),
+            };
+
+            // destroy
+            ffi::rocksdb_column_family_metadata_destroy(ptr);
+
+            // return
+            metadata
+        }
+    }
+
     /// Returns a list of all table files with their level, start key
     /// and end key
     pub fn live_files(&self) -> Result<Vec<LiveFile>, Error> {
@@ -2179,6 +2244,18 @@ impl<T: ThreadMode, I: DBInner> fmt::Debug for DBCommon<T, I> {
     }
 }
 
+/// The metadata that describes a column family.
+#[derive(Debug, Clone)]
+pub struct ColumnFamilyMetaData {
+    // The size of this column family in bytes, which is equal to the sum of
+    // the file size of its "levels".
+    pub size: u64,
+    // The name of the column family.
+    pub name: String,
+    // The number of files in this column family.
+    pub file_count: usize,
+}
+
 /// The metadata that describes a SST file
 #[derive(Debug, Clone)]
 pub struct LiveFile {