gguf: calculate tensor data offset

[drazdra]

adding tensors offset within file. offsets stored in metadata are relative to this offset and it's absent in metadata. with this field it's possible to actually access layers in gguf. without of it - offsets shown are useless.

[mishig25]

@drazdra happy to merge after:

1. linting issues are solved
2. tensorsOffset is added to https://github.com/huggingface/huggingface.js/blob/main/packages/gguf/src/gguf.spec.ts

[julien-c]

does this look valid @ngxson @Vaibhavs10?

[mishig25]

with this field it's possible to actually access layers in gguf. without of it - offsets shown are useless.

@ngxson is this correct?

[drazdra]

sorry about errors, i don't know TS, in my case i just patched the ready file in dist with JS and used it as is.

[ngxson]

This will introduce a breaking change for anyone who are using this tensorOffset field. I'd suggest adding a new field for it, for example tensorOffsetInFile. I'll make a PR for that.

[ngxson]

I changed the naming to tensorDataOffset instead of tensorsOffset, because it will be more aligned to gguf_get_data_offset(). Here is the equivalent cpp code:

int main(int argc, const char ** argv) {
    struct ggml_context * ctx_meta = NULL;
    struct gguf_init_params gguf_params = {
        /*.no_alloc = */ true,
        /*.ctx      = */ &ctx_meta,
    };
    auto * ctx_gguf = gguf_init_from_file("/Users/ngxson/Downloads/llama-2-7b-chat.Q2_K.gguf", gguf_params);
    printf("gguf_get_data_offset: %zu\n", gguf_get_data_offset(ctx_gguf));
    return 0;
}

Note sure why node test failed, I'll have a look later (I'm using a slow network)

[ngxson]

Should be ok now, matched the output from cpp code in my last comment.

Pinging @mishig25 for an extra review.