Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support other compression methods for squashfs #14

Conversation

xoofx
Copy link

@xoofx xoofx commented Aug 29, 2024

Fixes #13

Somes notes about this PR:

  • I have added the usage of https://github.com/Sergio0694/PolySharp to allow to use more recent language features. It doesn't add runtime dependencies but allows to use e.g init properties.
  • I have used Func<X,Y> for resolving compressors/streams to simplify the code and avoid polluting the namespace with delegates. If you prefer delegates for consistency of the codebase, let me know and I can change it.
  • I have kept the set of 0xDA (Best compression) when compressing with Zlib. It caused me some trouble because I forgot to remove it for the other compression. 😅 But not sure why it is there.

@xoofx
Copy link
Author

xoofx commented Aug 29, 2024

Testing with unsquashfs and I'm getting an error. It seems that I need to output a compressor options structure to the archive. Will dig into it.

@xoofx
Copy link
Author

xoofx commented Aug 29, 2024

Ok, so, it seems that it will require a bit more work than expected, as it requires to handle compression options (e.g for Lz4 here), though I thought that it was requiring to set a flags in the super block, but it seems that it does try to extract the options anyway (here)

It will take a bit more time to check all these details 😅

@LTRData
Copy link
Owner

LTRData commented Aug 30, 2024

Thanks a lot for working on this!

@xoofx
Copy link
Author

xoofx commented Sep 1, 2024

Ok, I got it working. Now I'm going to have to cleanup and optimize a bit more things.

I have one question: I need to add the implementation of a e.g FastMemoryStream derived from System.IO.Stream that I would like to use from the SquashFs project. Where would you prefer such implementation to live?

  • In DiscUtils.Streams? But it creates a new dependency to this project from squashfs...
  • In DiscUtils.Core using or not the namespace DiscUtils.Streams?
  • In DiscUtils.SquashFs?

@xoofx
Copy link
Author

xoofx commented Sep 1, 2024

Nevermind, I can actually workaround it with a standard MemoryStream, let's see!

@xoofx
Copy link
Author

xoofx commented Sep 1, 2024

Last commit should fix remaining issues. I have slightly optimized things related to usage of MemoryStream to reuse a shared one in the builder.

I have also successfully tested on a Linux box:

  • Mounting a squashfs archive created with DiscUtils with both zlib and lz4 compression
  • Using unsquashfs tools to stat/list the content of the archive

@LTRData
Copy link
Owner

LTRData commented Sep 1, 2024

Ok, I got it working. Now I'm going to have to cleanup and optimize a bit more things.

I have one question: I need to add the implementation of a e.g FastMemoryStream derived from System.IO.Stream that I would like to use from the SquashFs project. Where would you prefer such implementation to live?

  • In DiscUtils.Streams? But it creates a new dependency to this project from squashfs...
  • In DiscUtils.Core using or not the namespace DiscUtils.Streams?
  • In DiscUtils.SquashFs?

New Stream derived classes and other I/O related general purpose classes should be in DiscUtils.Streams. It would not create a new dependency, all DiscUtils projects use DiscUtils.Streams already. Some do not reference it directly though because it is brought in automatically by the dependency on DiscUtils.Core which uses DiscUtils.Streams.

@LTRData
Copy link
Owner

LTRData commented Sep 1, 2024

Great! Thanks a lot for your contribution! I'll take a look in the next few days!

@LTRData
Copy link
Owner

LTRData commented Sep 2, 2024

I have had some quick looks at the code. It looks good for most parts, but just a couple of things:

  • Looking at the diff for .gitignore, I am fine with the added things, but not the removed things. We need to keep things like launchsettings.json and certain build artifacts out of the repository.

  • Test project, I would like to still support .NET Framework 4.6. This requires version 1.2.16 of K4os.Compression.LZ4 that you added. This requires some changes to the project file to bring in different versions for different target frameworks.

I can branch the pull request and make some changes that I have in mind!

@LTRData LTRData changed the base branch from LTRData.DiscUtils-initial to add-support-other-compression-methods-for-squashfs September 2, 2024 22:14
@LTRData LTRData merged commit 0f41198 into LTRData:add-support-other-compression-methods-for-squashfs Sep 2, 2024
1 check passed
@LTRData
Copy link
Owner

LTRData commented Sep 2, 2024

I created a new branch for working on this until I merge it into master branch.

@xoofx
Copy link
Author

xoofx commented Sep 3, 2024

Thank you for taking this PR, looking forward for the final merge! ☺️

LTRData added a commit that referenced this pull request Sep 8, 2024
* Update .gitignore with standard VS/Resharper and various stuffs

* Add PolySharp to support init properties

* Add support for other compression methods to squashfs

* Fix options to only emit them if necessary. Optimize MemoryStream usage for compressed streams.

* Add support other compression methods for squashfs (#14)

* Update .gitignore with standard VS/Resharper and various stuffs

* Add PolySharp to support init properties

* Add support for other compression methods to squashfs

* Fix options to only emit them if necessary. Optimize MemoryStream usage for compressed streams.

* Fixed/reverted some changes

* Corrected mistake in target framework versions

---------

Co-authored-by: Alexandre Mutel <alexandre_mutel@live.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for pluggable compression/decompression for Squashfs?
2 participants