Alarming memory use and non-termination.

[frankmcsherry]

I cloned the rust serialization benchmark repository, which I have been able to cargo bench. I made a modification, to use a local checkout of the columnar crate, with this diff

mcsherry@gallustrate rust_serialization_benchmark % git diff
diff --git a/Cargo.toml b/Cargo.toml
index 9936b28..6bb79e3 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -55,7 +55,8 @@ cbor4ii = { version = "=1.0.0", features = [
     "serde1",
 ], optional = true }
 ciborium = { version = "=0.2.2", optional = true }
-columnar = { version = "=0.11", optional = true }
+#columnar = { version = "=0.11", optional = true }
+columnar = { path = "../columnar", optional = true }
 databuf = { version = "=0.5.0", optional = true }
 dlhn = { version = "=0.1.7", optional = true }
 flatbuffers = { version = "=25.12.19", optional = true }
mcsherry@gallustrate rust_serialization_benchmark % 

With the local checkout unchanged, the following seems to build and run:

cargo bench --no-default-features --features columnar

With a light change to the columnar project, at least I think light:

mcsherry@gallustrate columnar % git diff
diff --git a/src/lib.rs b/src/lib.rs
index 09189ca..d6ff059 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -872,17 +872,17 @@ pub mod bytes {
         pub fn decode(store: &[u64]) -> impl Iterator<Item=&[u8]> {
             assert!(store[0] % 8 == 0);
             let slices = (store[0] / 8) - 1;
-            (0 .. slices).map(|i| decode_index(store, i))
+            let bytes: &[u8] = bytemuck::cast_slice(store);
+            (0 .. slices).map(|i| decode_index(store, bytes, i))
         }
 
         /// Decodes a specific byte slice by index. It will be `u64` aligned.
         #[inline(always)]
-        pub fn decode_index(store: &[u64], index: u64) -> &[u8] {
+        pub fn decode_index<'a>(store: &'a [u64], bytes: &'a [u8], index: u64) -> &'a [u8] {
             debug_assert!(index + 1 < store[0]/8);
             let index: usize = index.try_into().unwrap();
             let lower: usize = ((store[index] + 7) & !7).try_into().unwrap();
             let upper: usize = (store[index + 1]).try_into().unwrap();
-            let bytes: &[u8] = bytemuck::try_cast_slice(store).expect("&[u64] should convert to &[u8]");
             &bytes[lower .. upper]
         }
 
mcsherry@gallustrate columnar % 

things seems to no longer work, and the same cargo --bench command does not get past rustc, which runs for quite a long time and uses seemingly unbounded memory. I terminated it around 85GB.

It's possible I'm totally missing something obvious, but also the non-terminating rustc and 85GB footprint doesn't seem right either.

Meta

The reported version is

mcsherry@gallustrate rust_serialization_benchmark % rustc --version
rustc 1.94.0-nightly (2f1bd3f37 2026-01-12)

which I think might be a function of the repository. I then set it to stable and things don't build, so the nightly choice is probably intentional. It's possible that the issue is restricted to nightly!

[lqd]

The repo uses nightly, how were you able to say this is a stable regression @theemathas?

[theemathas]

My bad, I misread the issue report.

[apiraino]

@frankmcsherry as I see many moving parts, do you think you can try reducing a bit so we can attempt a bisection (we usually use cargo-bisect-rustc).

thanks

[frankmcsherry]

I'm able to box up the instructions, but one of the repos is foreign to me (rust_serialization_benchmark) and is def confusing. My guess is that I'll be able to get instructions that look like

git clone <rust_serialization_benchmark>
git clone <columnar>
<edit rust_serialization_benchmark/Cargo.toml>
<edit columnar/lib.rs>

at which point the presence/absence of the last step seems to flip the bit between works/doesn't. I'm not sure I'll be able to dive deeper into stripping away large blocks of code from either of the projects though. To make things worse, there's also a columnar_derive that backs the derive(Columnar) statements.

So, I'll get the above as what "reproduces" for me. Ideally that should lead to a bisection, at least if it goes from "working" a while to "not working" now. The edits are already as minimized as I know how to make them, though.

[frankmcsherry]

The following reproduces for me:

mkdir repro
cd repro
git clone git@github.com:djkoloski/rust_serialization_benchmark.git
git clone git@github.com:frankmcsherry/columnar.git
cd rust_serialization_benchmark
hx Cargo.toml
    <apply that diff above>
cargo bench --no-default-features --features columnar

This builds 108 things, uses between 1 and 3 GB of memory, and finishes the compilation for me at least after this long:

    Finished `bench` profile [optimized] target(s) in 4m 38s

The repro then continues

hx ../columnar/src/lib.rs
    <apply the second diff above>
cargo bench --no-default-features --features columnar

This is at 24GB as I type this, has been running for seven minutes, and I expect to continue to grow like it has previously done, but unlike the cargo bench command just a few lines earlier in the repro.