Failed to deserialize

[leontoeides]

Good morning,

Thank you for this crate! It's exactly what I've been looking for. The way you've handled secondary indicies is perfect 👌and very powerful.

Unfortunately when I use the crate I get this error:

Failed to deserialize the struct #struct_name
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This happens when I try to scan the entire database.

        rtxn
            .scan()
            .primary()?
            .all()
            .for_each(|my_type: MyType| { /* my code here */ });

I might be able to provide more details later, I just wanted to make sure I opened this issue before I forgot

[vincent-herlemont]

@leontoeides Thank you for reporting this issue. The error message is unclear, and a first improvement has just been merged #140. And secondly, instead of causing the program to crash, a proper error will be returned #141.

Now related to your issue, in my opinion, you are storing data in your database and evolving your models without splitting them into different versions. This means that you are trying to query a model that no longer has the same fields or the same types of fields. This can often happen when you are developing an application. If you delete your database and re-run your code, does the problem still occur?

[leontoeides]

I'll try to provide more details, eventually, when I can.

In my case, the struct definition has not changed. I was working on a brand new blank database, and I was reading back all records I'd just written in the same execution.

The struct I'm using is pretty hefty and I'd imagine it'd be a challenging case for a serializer/deserializer. I am successfully using this (almost) same struct with rmp-serde in a different project, though. The struct includes others structs and multiple collections. ~40 records weighed in at about 1.5 Mb in the file.

[vincent-herlemont]

@leontoeides Ok, so under the hood, native_db uses native_model to serialize/deserialize these types. By default, native_model uses bincode/1.3.3 with serde. In order to see where the problem come from, can you please perform the following tests:

• Try to encode/decode your data without native_db using native_model’s encode / decode
• Try to encode/decode your data with bincode/1.3.3 and serde.
  And see if there is an error.

If the error persists and you find no solution, you can always use rmp-serde with native_model by using the custom serialization/deserialization methods described here: native_model/Custom serialization format.

[leontoeides]

Both native_model and bincode failed to round-trip my struct.

bincode's error was DeserializeAnyNotSupported or Bincode does not support the serde::Deserializer::deserialize_any method. This error occurs in both versions of bincode.

I remember this one. I recall playing with bincode before. I had to move away from it because it doesn't work with my data. You may want to do the same (move away from using bincode as a default) for this same reason.

musli::storage looks interesting but there's no serde support.

Also, postcard failed with DecodeBodyError(DecodeError { msg: "This is a feature that PostCard will never implement", source: This is a feature that PostCard will never implement

Just some feedback - the struct for postcard was hard to find because it didn't make it into the docs. It seems like there's a way to document featured-gated items, though.

This was tricky to find:
#[native_model(id = 1, version = 1, with = native_model::postcard_1_0::PostCard)]

I implemented a codec using rmp-serde (see below) and it works well for me. The encoder/decoder trait + macro setup is very clever! Your crate is full of surprises. It seems like you've thought of everything.

pub struct RmpSerde;

impl<T: serde::Serialize> native_model::Encode<T> for RmpSerde {
    type Error = rmp_serde::encode::Error;
    fn encode(obj: &T) -> Result<Vec<u8>, Self::Error> {
        rmp_serde::encode::to_vec(obj)
    }
}

impl<T: for<'de> serde::Deserialize<'de>> native_model::Decode<T> for RmpSerde {
    type Error = rmp_serde::decode::Error;
    fn decode(data: Vec<u8>) -> Result<T, Self::Error> {
        rmp_serde::decode::from_slice(&data)
    }
}

[vincent-herlemont]

Just some feedback - the struct for postcard was hard to find because it didn't make it into the docs. It seems like there's a way to rust-lang/cargo#8103, though.

Thanks to pointing it, I have opened an issue here vincent-herlemont/native_model#70.

---

@leontoeides It might be a good idea to add rmp-serde as another available decoder with a feature to native_model. What do you think? Feel free to make a PR if you wish.

---

@leontoeides Is everything good for you on this topic? If so, I will let you close the issue.

[LunaeSomnia]

Hi. I'm also experiencing this kind of issues in my project. @leontoeides, could you specify what exactly you did to fix the issue on your end? Thanks.

[leontoeides]

Hi. I'm also experiencing this kind of issues in my project. @leontoeides, could you specify what exactly you did to fix the issue on your end? Thanks.

Hello @LunaeSomnia, it looks like the default serializer bincode isn't feature complete so it's crashing. Try using rmp-serde instead, which is a good library.

1. Include rmp-serde in your project.
2. Include the following struct and trait implementations in your project. It tells native_db how to encode/decode your struct to/from bytes when writing to disk (or memory)

pub struct RmpSerde;

impl<T: serde::Serialize> native_model::Encode<T> for RmpSerde {
    type Error = rmp_serde::encode::Error;
    fn encode(obj: &T) -> Result<Vec<u8>, Self::Error> {
        rmp_serde::encode::to_vec(obj)
    }
}

impl<T: for<'de> serde::Deserialize<'de>> native_model::Decode<T> for RmpSerde {
    type Error = rmp_serde::decode::Error;
    fn decode(data: Vec<u8>) -> Result<T, Self::Error> {
        rmp_serde::decode::from_slice(&data)
    }
}

3. Apply the following derive macro to the struct you'd like to use with native_db. This tells native-db to use rmp-serde instead of the default bincode serializer.

#[native_model(id = 1, version = 1, with = RmpSerde)]
struct My Struct {
    pub string: String,
}

@vincent-herlemont Yes I will try to make a PR sometime this weekend

[LunaeSomnia]

Hey @leontoeides, I just implemented your instructions & ended up working. The serialization made by rmp-serde is surprisingly quick and I'll be using it from now on in my project.

Thank you very much for your time.

[vincent-herlemont]

With the merging of this PR #141, the database will no longer cause a panic related to deserialization. Each query returns a result. This will be available in version 0.7.0 of native_db.