WASM support?

[bluenote10]

First of all, thanks for maintaining this library, this is quite awesome!

I was wondering if it would be feasible to officially support wasm? Ignoring zip support, it looks like it might be doable, but I'm not sure how much effort it really is. I made this minimal experiment:

fn main() {
    let data = include_bytes!("test.npy").as_slice();
    let npy = npyz::NpyFile::new(data).unwrap();
    println!("{} {:?}", npy.len(), npy.shape());
}

with a tiny test.npy generated via:

import numpy as np
np.save("test.npy", np.array([1, 2, 3]))

Trying to compile it with cargo build --target wasm32-unknown-unknown currently fails to compile:

514 |     const SAFE_U16_CUTOFF: usize = 0x1_0000_0000 - 0x400;
    |                                    ^^^^^^^^^^^^^^^^^^^^^ attempt to compute `0_usize - 1024_usize`, which would overflow

Simply replacing the expression in line

npyz/src/write.rs

Line 514 in 2e08278

const SAFE_U16_CUTOFF: usize = 0x1_0000_0000 - 0x400;

by 0xffff_fc00 seems to fix that issue and the compilation now succeeds.

I'm a bit unsure if this is all that is needed to properly support wasm though. Runtime testing is a bit tedious with wasm apparently. I extended the minimal example to become executable under nodejs by using wasm-pack test

#[wasm_bindgen_test::wasm_bindgen_test]
fn test_wasm() {
    console_error_panic_hook::set_once();

    let data = include_bytes!("test.npy").as_slice();
    let npy = npyz::NpyFile::new(data).unwrap();
    web_log::println!("{} {:?}", npy.len(), npy.shape());

    assert_eq!(npy.len(), 3);
    assert_eq!(npy.shape(), [3]);

    let values: Vec<_> = npy
        .data::<i64>()
        .unwrap()
        .into_iter()
        .map(|value| value.unwrap())
        .collect();

    assert_eq!(values, vec![1, 2, 3]);
}

with appropriate dependencies added to Cargo.toml:

[dependencies]

console_error_panic_hook = "0.1.7"
npyz = {path = "<path-to-my-fork>", features = []}
wasm-bindgen = "0.2.86"
wasm-bindgen-test = "0.3.36"
web-log = "1.0.1"

Running wasm-pack test --node executes the test successfully! But admittedly that is a very tiny bit of functionality tested. It would be interesting to actually run the existing tests against wasm, but that seems tricky, because to my knowledge the regular #[test] tests cannot be easily executed in wasm, and duplicating all tests for tagging them with #[wasm_bindgen_test::wasm_bindgen_test] is also a no go.

Any thoughts on how to proceed? I could open a PR to fix the compile time support, but I'm still a bit uncertain if it actually works and how to verify it systematically...

[ExpHP]

Status: With #66 and #70, the following cargo features are supported/tested:

• derive
• arrayvec
• complex
• npz

[ExpHP]

Sadly, neither of these seem to work as expected for running the unit tests in the library, which would be the ideal way to test complex and arrayvec:

$ wasm-pack test --node --lib --features complex,arrayvec
$ wasm-pack test --node -- --lib --features complex,arrayvec

both trivially succeed with

[INFO]: 🎯  Checking for the Wasm target...
    Finished dev [unoptimized + debuginfo] target(s) in 0.02s
[INFO]: ⬇️  Installing wasm-bindgen...
    Finished test [unoptimized + debuginfo] target(s) in 0.02s
     Running unittests src/lib.rs (target/wasm32-unknown-unknown/debug/deps/npyz-3581acb1995a1aa6.wasm)
no tests to run!

I found this issue which claims that these flags actually make wasm-pack panic (though it's an older version): drager/wasm-pack#1289

[ExpHP]

.....it just occurred to me that the reason it probably found no tests is because the lib isn't using the #[wasm_bindgen_test] attribute. Oops. Hang on.

Edit: Ew, I would also need to make the module hierarchy public. Alright, yeah, I'm really not sure what the best practices are here.

[bluenote10]

I would also need to make the module hierarchy public.

Yes, exactly that limitation feels pretty severe currently, and I also couldn't find any decent work-around. It should be noted that the wasm book has a section on both adding wasm support and maintaining wasm support, and but they mainly reduce the problem to building for wasm. This suggests that a successful wasm build in combination with successful native tests may have a high chance of actually working at runtime on wasm as well? I have no clue how strong the "it compiles and it works for a different target" actually is, but I could imagine that in particular for a parsing library that operates on binary level, actual runtime checks would have been nice.

Thanks for the release in any case, and no worries regarding the delay (there's no hurry 😉)!