[FR] On-device decompression for saving time on slow wifi transfers

[vlsi]

Is your feature request related to a problem? Please describe.

I want sending a compressed gcode over wifi, and I want decompress the file before printing.
It would reduce wifi transfer times.

Are you looking for hardware support?

No response

Describe the feature you want

I need a gcode for decompressing files on board. I do not need folder management, when unzipping, and one input one output would be enough

Additional context

I use mks nano robin 3.2 board, and wifi seems to be limited to 115kbps there, however, the CPU is 178MHz. I expect zstd decompressor to work reasonably well on that board.

Having a second pass over the received gcode might help for verifying the input as well (see #26016)

[ellensp]

Marlin already has meatpack and binary file transfer both of which have compressed gcode.

This issue is that esp3d does not support either of these (as far as im aware)

[vlsi]

I never heard of meatpack.
I just checked with https://www.thingiverse.com/thing:5384951

57552	articulated_axolotl_2.stl
66580	axolotl.gcode
34620	axolotl.gcode.meatpack
24380	axolotl.gcode.zst

Of course, having meatpack is better than nothing, however, just using ztds enables having even better compression with saving all the comments (e.g. mks preview images within the gcode).

I thought wifi_upload should not care of the encoding as it sends the file with HTTP post (see https://github.com/Jeredian/mks-wifi-plugin/blob/17103b3f9dbbf17ba27cb255c98b15693115d25d/MKSOutputDevice.py#L514-L520), and the device stores it as a file.

WDYT of adding a gcode for decompressing a file like M... C. in_file out_file? (C would stand for codec which might be C0 for meatpack)

[ellensp]

Both the current systems don't create a compressed file
They use compressed transport layers to upload faster, but the resulting file is still a stock gcode file.

I don't think a .z gcode files would work, marlin needs to be able to skip around the file, difficult to do while compressed.

[vlsi]

I don't think a .z gcode files would work, marlin needs to be able to skip around the file, difficult to do while compressed.

I mean receive compressed file, then decompress, and print decompressed file only. In other words, I do not suggest printing from a compressed file.

[ellensp]

Prusa has purposed bgcode https://github.com/prusa3d/libbgcode

[vlsi]

Thanks for the pointer.
I've created several issues there:

• Print metadata: add standard keys like "Overall dimensions", "first layer dimensions", "total filament length/weight" prusa3d/libbgcode#28
• WDYT of using key-value encoding rather than INI encoding? prusa3d/libbgcode#27
• Use an existing serializer to define the format in a machine-readable way: protobuf, flatbuffers, ... prusa3d/libbgcode#29

[vlsi]

It is sad prusa3d/libbgcode use a mix of INI and binary encoding. It would be so much better if they went with a protobuf or a similar schema format.

[gudvinr]

It would be so much better if they went with a protobuf or a similar schema format.

Now try to imagine how it would work.

Having schema would require all fields to be supported by all slicers and all firmwares.

Then, you need some kind of committee to agree on whoch options to include in schema. Good luck having all manufacturers to agree on something.

Okay. Then you need some kind of extension system that makes it possible to include fields supported by manufacturers.
Now you have file format that can have arbitrary fields with arbitrary values. Which INI already is.

Having schema would not bring anything to the table and would be impossible to actually use.

Binary encoding? Maybe. But its only benefit is compact size and it's irrelevant for options section because it's very small compared to gcode which is compressed. And with cbor you will have keys and values encoded as strings anyway.

[vlsi]

Now try to imagine how it would work.
Having schema would require all fields to be supported by all slicers and all firmware.

Not really. In proto3 all fields are optional, so they are replaced with null or the corresponding default value depending on the schema. It does not force producers and consumers to generate or recognize "unknown" fields.

Now you have file format that can have arbitrary fields with arbitrary values. Which INI already is.

INI lacks data types.

Having schema would not bring anything to the table and would be impossible to actually use.

A common extension to g-code is a blob with an image that might better be shared and declared similarly across firmwares.

See prusa3d/libbgcode#28.
Here are samples of the common metadata that might be helpful:

• Overall dimensions. The printer might reject or warn the user in case the printed model will not fit into the print volume
• Dimensions of the "first layer". It might be helpful so the printer could probe the relevant bed surface with a probe. It could reduce the probing time and, it could improve the quality of the bed mesh
• Total filament length for the print. It might be helpful on the info screen before user accepts the print

Sure "total filament length" is irrelevant for laser cutters, however, well, the fields are optional.

Binary encoding? Maybe. But its only benefit is compact size and it's irrelevant for options section because

Binary encoding is easier to parse.

[gudvinr]

In proto3 all fields are optional

It's not the point. Point is that you need to have schema supported and updated everywhere.
We don't live in a magic world where schema with all available optional fields just exists and everyone uses it.

In protobuf, fileds defined with field number, not even symbolic keywords.

Even if by some miracle there will be some "default" set of fields, at some point manufacturers will need to extend format. No one will be held responsible to follow agreements on how to avoid conflicts.

You'll simply end up with a mess of incompatible bgcodes where manufacturer A encodes field 34 as integer for flow control and manufacturer B encodes field 34 as text for object name or whatever.

They most likely will not publish updated schemas too. To support all this bs in every firmware you need to make workarounds. OR in all slicers you will need to make additional "flavors" so produced gcodes won't fail.
This probably won't happen so compatibility between slicers and firmwares will simply shrink significantly.

And one important thing is that gcode by design is not printer specific format, so any kind of printer-centric schema basically bolts down file format to be used for printers only.

Here are samples of the common metadata that might be helpful:

In this issue I also listed why it's impractical so I won't repeat myself here too.

You need to understand that bgcode is basically just a container. And you're trying to "enhance" it by limiting its ability to contain stuff.

Binary encoding is easier to parse.

INI isn't hard to parse by any means. There are tons of INI parses available too.
It's very moot point because you only need to write parser so there's very little overhead for that.

It may not be the best choice. But it's good enough choice.
And compared to "better" options it is already written and already implemented by someone. And "better" options aren't. This alone makes this more useful than imaginary "better" solutions.

[vlsi]

You'll simply end up with a mess of incompatible bgcodes where manufacturer A encodes field 34 as integer for flow control and manufacturer B encodes field 34 as text for object name or whatever.

You are 100% right. The numeric fields in the schema should be declared by someone.
Of course, if people randomly extend the schema with their ids, then the ids might collide.

There's a lightweight option to have extra fields in the schema:

map<string, string> string_props = 3;
map<string, int> int_props = 4;

Let's imagine there was a GitHub repository 3d-printer-file-spec/3d-printer-file-spec that included a canonical schema.
Then it could be a common proto for everybody. See https://github.com/sigstore/protobuf-specs/tree/main/protos as an example.

limiting its ability to contain stuff.

Could you please elaborate? Since schema fields are optional, the other device types can have their metadata: there might be a metadata section for laser cutters, a metadata section for CNC, and so on. I do not quite understand why you mention "limiting its ability"

[gudvinr]

Not IF, people absolutely will do that. You're trying to imagine perfect world where everyone follows agreements and everything just works. It's not like that in the real world.