Add support for loading and exporting large OBJ files #16271
Conversation
|
Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s). |
|
Snapshot stored with reference name: Test environment: To test a playground add it to the URL, for example: https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/16271/merge/index.html#WGZLGJ#4600 Links to test babylon tools with this snapshot: https://playground.babylonjs.com/?snapshot=refs/pull/16271/merge To test the snapshot in the playground with a playground ID add it after the snapshot query string: https://playground.babylonjs.com/?snapshot=refs/pull/16271/merge#BCU1XR#0 |
|
WebGL2 visualization test reporter: |
|
Visualization tests for WebGPU |
|
Hi! First of all, thanks a lot for the PR! We are currently in a code freeze, aiming to release babylon 8.0 in a few weeks. We let bug fixes and feature additions in, but some PRs will not be merged, even if approved, until 8.0 is released. Just a heads up. Regarding this PR - I am not sure this is something we want to support directly in our obj file loader. I can see it implemented at a higher level, maybe, to, in general, support compressed filed, but we will have to be very careful with what is being used, making sure all of our supported environments (including babylon native) supports these features. This can always be an external OBJ File loader that is added to the loaders list from a different package. I personally love the idea of making babylon as extendable as possible, and I believe this kind of loaded is a perfect example for it. Anyhow, specifically to your PR - we don't fetch directly using |
|
Hi! It's not just about supporting compressed files, but rather about fixing the bottleneck caused by the fact that the function loading OBJ files in Babylon first loads the entire file into a string before parsing it and creating the meshes. This causes the browser to crash when trying to load a file larger than 50MB. I'm not an expert in the Babylon repository, so I'm not sure where this should be integrated at a higher level. However, without this functionality, Babylon.js is limited to loading only small files and consumes more memory than necessary. Is there a mechanism using the File Loader tool that allows reading the file in chunks and processing it progressively as it is being read? |
Nope, there isn't. This is why I suggested this might need to go somewhere else. It could be added as an optional parameter to the LoadFile function to return the data in chunks (maybe using the onProgress callback) instead of the entire data in the onSuccess callback. |
|
Moving this to draft to continue the discussion. |
| reader = response.body.getReader(); | ||
| total = parseInt(response.headers.get("Content-Length") ?? ""); | ||
| } else { | ||
| const ds = new DecompressionStream("gzip"); |
There was a problem hiding this comment.
A note here, DecompressionStream api is relatively new, fallbacks might be needed.
There was a problem hiding this comment.
I haven't noticed any issue on the official documentation https://developer.mozilla.org/en-US/docs/Web/API/DecompressionStream; every browser is green checked.
What fallback do you suggest ? is it acceptable to include an external lib such as pako ?
How can we go ahead with this ? |
In any way, even if approved, this will not be merged until 8.0 is released. IMO - the best way to move forward is to release an external OBJ loader that uses this mechanism and provide it as a standalone package. It can be replaced with the babylon OBJ loader for people who requires this. There are a few reasons to that - First, 8.0 release. I don't mean to be a downer here, I actually really like your idea. It is just not as simple as changing the OBJ loader. there is a lot to it. We will be happy to look into it, based on this PR, if you want to submit a feature request on the forum! Otherwise, really, the simplest would be an external package that any user can consume. |
This pull request introduces support for loading and exporting large OBJ files without requiring the entire file to be loaded into memory as a single string.
Loading Enhancements
The default behavior remains unchanged when using BABYLON.SceneLoader, ensuring full backward compatibility.
However, SceneLoader still fetches the entire file, as I could not find a way to prevent this.
To leverage an incremental reading and building mechanism, the OBJFileLoader class should be used directly, importing meshes without the data parameter:
This new loading mechanism supports both .obj and .obj.gz files, allowing seamless handling of compressed OBJ files.
Parsing Fixes
Fixed an issue where some OBJ files with multiple space-separated values (e.g., f 1 4 5) were incorrectly parsed.
The new behavior correctly normalizes them to f 1 4 5, addressing a case that was not handled by the existing solidParser string manipulation methods.
Exporting Changes (Breaking Change)
The export method has been modified to return an iterator instead of a full OBJ string.
This change aligns with the new incremental approach, allowing the data to be processed in chunks instead of requiring full memory allocation.
The new export method is not backward-compatible but enables efficient writing via a stream writer to output the data progressively to a file or another destination.
For those who still need the full OBJ as a string, it can easily be converted using:
const obj = [...BABYLON.OBJExport.OBJ(meshes)].join("\n");