Implements qs2 serialization as an option

[shikokuchuo]

Closes shikokuchuo/mirai#162.

Adds 'qs2' as an option for all send/recv functions.

Requires the qs2 package with registered C callables reaching CRAN.

[shikokuchuo]

@traversc I'm using the parameter defaults you currently have in your header file (without actually using your header file). Let me know if you think something else might be optimal here. Thanks!

[traversc]

Default parameters are safe, but optimal depends on use case. I think it would be nice to expose the parameters somehow (or at least the compression parameter). Could be a future update.

[shikokuchuo]

The interface would get really busy. An alternative I'm thinking about is having a global switch that swaps out R serialization with qs, and that function could then possibly set the parameters.

[shikokuchuo]

Scratch that idea with the global parameter - it doesn't sit well with the design of nanonext. On the other hand, if there's a way for qs2 to set default parameters independently, then I'd be happy to use the default rather than a fixed setting.

[traversc]

By design, do you mean user interface design?

In which case I can implement global parameters in the qs2 package and you could pull from there.

Another idea, you have send_mode and recv_mode parameters -- do these need to only accept strings/integers? For example, you could also accept a list(mode = "blah", ...) where ... are additional named parameters passed to the method for mode "blah".

Just an idea, totally understand if it doesnt jive with your preference.

[shikokuchuo]

Hi @traversc where I think I'm coming out at the moment is I would want to use qs2 serialization automatically instead of R serialization if qs2 is already loaded. This would avoid any changes to the UI entirely.

So I have the following questions:

• Does qs2 begin with a certain byte (sequence) so I can easily detect it is qs2-encoded? I think I need this for mirai as in a new process, qs2 won't be loaded, but if the encoded message is detected as qs2 then it can be loaded automatically.
• My goal is now purely speed, so do you have benchmarks at the lowest compression rate?

Thanks!

[traversc]

Yes, there is a 4 byte sequence: https://github.com/qsbase/qs2/blob/main/src/qx_file_headers.h#L19

Here are some benchmarks and others in that repo: https://raw.githubusercontent.com/qsbase/qs2_analysis/main/benchmarks/plots/ubuntu_write_benchmarks.png

The point on the far left is the compression level = 1.

What do you mean by focusing on purely speed? Does this account for network transfer times? If it is only serialization to memory then R_serialize without any compression will be faster.

[shikokuchuo]

Sounds good! I'll put together an implementation on this basis.

I may be getting ahead of myself with the speed comment. Compression may actually still be useful in memory-limited environments. I think I'll just approach this empirically.

If qs2 eventually offers a way to set a global default, then I'll move it to use that.

[traversc]

Regarding empirical benchmarking, it highly depends on the dataset. The repo I linked to above collates 20 small to medium size datasets 100 MB ~ 10 GB in size that you could use.

I went ahead and implemented global options: https://github.com/qsbase/qs2?tab=readme-ov-file#global-options-for-qs2

If that approach looks good, I can send it to CRAN.

[shikokuchuo]

Thanks @traversc I just took a quick look at the global options, and this wasn't entirely what I had in mind.

1. I think you'll agree getting and setting R options is slow.
2. I'd want to access them from C, so I was thinking a value of e.g. '0' to the relevant parameter would pick up the default.
3. So, in terms of implementation, perhaps have some variables in global scope, and custom getter and setter functions for the R level.

I've only been able to look at this quickly, so if there's an issue you're encountering, just let me know.

Btw. this isn't essential and won't be blocking - I can choose a default for the initial implementation. So please take your time on this.

[traversc]

I didn't want to use special values like 0 to mean default, so here's what I came up with in the latest commit.

From the R side, a user can set global options with e.g. qopt("compress_level", 4). This calls a C++ function that sets a variable with static scope. Any function on the R side that uses compress_level will now use a value of 4.

From the C-side, you can retrieve this global option with qs2_get_compress_level() and similar functions for other global options. Alternatively I can expose a new function like qs_serialize_default that uses default parameters.

Here's a full Rcpp example:

#include <Rcpp.h>
#include "qs2_external.h"
// [[Rcpp::depends(qs2)]]
using namespace Rcpp;

// serialize to memory using global user-defined defaults
//
unsigned char * qs_serialize_default(SEXP object, size_t* len) {
  std::cout << "compress_level: " << qs2_get_compress_level() << std::endl;
  std::cout << "shuffle: " << qs2_get_shuffle() << std::endl;
  std::cout << "nthreads: " << qs2_get_nthreads() << std::endl;
  return c_qs_serialize(
    object,
    len,
    qs2_get_compress_level(),
    qs2_get_shuffle(),
    qs2_get_nthreads()
  );
}

// deserialize from a memory buffer using global user-defined defaults
//
SEXP qs_deserialize_default(const unsigned char * buffer, const size_t len) {
  return c_qs_deserialize(
    buffer,
    len, qs2_get_validate_checksum(),
    qs2_get_nthreads()
  );
}


// [[Rcpp::export]]
SEXP roundtrip(SEXP x) {
  size_t len;
  unsigned char * buffer = qs_serialize_default(x, &len);
  SEXP output = qs_deserialize_default(buffer, len);
  c_qs_free(buffer);
  return output;
}


/*** R
library(qs2)
qopt("compress_level", 1) # sets global default to 1
qopt("shuffle", FALSE) # sets global default to FALSE
qopt("nthreads", 4)

data <- runif(1e8)
result <- roundtrip(data)
identical(data, result)
*/

[shikokuchuo]

In reply to your question on the C level functions for getting the defaults - if there is any meaningful performance impact then I would obviously prefer a qs_serialize_default() function.

I've made progress on this - you can see what I have in mind from the current state of this PR. I'd still need to add tests, but mirai tests fine with nanonext::use_qs2() (I'd also need to think of a better name for that function 😅 ).

The choice of serialization function is determined by a global variable with the different paths implemented directly in nano_serialize and nano_unserialize().

Because of this, I am actually thinking it makes sense to incorporate straight into sakura. If I export C callables there then not only nanonext could use this, but any other package as well.

[traversc]

if there is any meaningful performance impact then I would obviously prefer a qs_serialize_default() function.

I dont think there would be a performance difference, I'd be pulling the global static variables the same way.

Because of this, I am actually thinking it makes sense to incorporate straight into sakura. If I export C callables there then not only nanonext could use this, but any other package as well.

Sakura currently has in memory serialize/unserialize, but the most likely use case would be writing to disk, right? If so ideally we would want to stream to disk, rather than serialize to memory and write to disk all at once. Otherwise we would be leaving performance on the table and also requiring higher memory usage.

[shikokuchuo]

Leaving aside the global options for now, I started testing, and I'm getting something like this with qs2_serialize(object, &buf->cur, 1, true, 1) which uses the lowest compression level.

I'm choosing to ship a largish object so that most of the time is spent on serialization. Is this roughly what you would expect?

x <- rnorm(1e6)
library(mirai)
daemons(8)
#> [1] 8
microbenchmark::microbenchmark(collect_mirai(mirai(x, x = x)))
#> Unit: milliseconds
#>                            expr      min       lq     mean   median       uq
#>  collect_mirai(mirai(x, x = x)) 14.86581 18.27487 21.88207 21.07917 23.89938
#>       max neval
#>  39.41691   100
nanonext::use_qs2()
#> NULL
microbenchmark::microbenchmark(collect_mirai(mirai(x, x = x)))
#> Unit: milliseconds
#>                            expr      min       lq     mean   median       uq
#>  collect_mirai(mirai(x, x = x)) 25.55989 28.87685 33.61875 30.80444 34.23783
#>       max neval
#>  124.0863   100

^{Created on 2025-03-03 with reprex v2.1.1}

[traversc]

Yes that looks reasonable. If you are limited by serialization (and not memory or transfer speed) then using qs2 is not beneficial.

But also you could try -1 .. -5 as compress level and adding more threads.

[shikokuchuo]

I'm sorry I've been under the impression that speed would be similar to base R with better compression built in i.e. it would strictly dominate. As there is a trade-off, then it seems that qs2 would really only make sense where memory is relatively constrained.

You may have seen that I've released sakura to CRAN. I am now thinking that this is going to be the best place to integrate qs2. As that package may have alternative uses, it may be more suited to qs2 and you'd also be free to add more functionality e.g. to write directly to files.

You would probably not implement the integration in exactly the same way - I'll leave it to you to consider. But please feel free to take anything in this PR over there. Also please don't let this discussion hold up a CRAN release on your end.

The intention is for nanonext to either directly import or vendor sakura going forward.

[shikokuchuo]

Really sorry for not merging this PR as is, but I think this will be better and cleaner in the long run.

Also I do want to encourage you to contribute to that package. I intend to move it to the RConsortium organization and I'd be very happy to include you as 'aut' upon a contribution! 😄

[traversc]

I'm sorry I've been under the impression that speed would be similar to base R with better compression built in i.e. it would strictly dominate. As there is a trade-off, then it seems that qs2 would really only make sense where memory is relatively constrained.

You can get closer by using negative compression levels and shuffle = FALSE e.g.

qs_serialize(x, compress_level = -5, shuffle = FALSE)

But for sure if you are not memory constrained/bandwidth constrained there is no value. For me, disk usage is important so I targeted compression levels around what saveRDS does by default.

You may have seen that I've released sakura to CRAN. I am now thinking that this is going to be the best place to integrate qs2. As that package may have alternative uses, it may be more suited to qs2 and you'd also be free to add more functionality e.g. to write directly to files.

That sounds reasonable. I will think on how best to integrate from the code perspective.