Create Ishan Darji GSoC Blog

[Ish-D]

Final blog for HSF 2024 GSoC project "Lossless compression of raw data for the ATLAS experiment at CERN".

[Ish-D]

I do not seem to have the permission to add reviewers, but pinging my mentor @maszyman for visibility.

[maszyman]

Thanks a lot @Ish-D for this nice blog post (and your work in general)!

Please have a look at a couple of my comments.

[maszyman]

Suggested change

Below is a table of data for compressors that were thought to be relevant to look at. One thing that is important to note is that much of the ATLAS experiment data has previously used zlib with compression level 1 for compression, so it will be treated as something of a baseline and the remaining libraries will be compared to it. Below are tables of libraries that were thought to be particularly notable, for each, we take note of the library used and what compression level, the ratio ((compressed size / original size) * 100), as well as compression and decompression speed.

Below is a table of data for compressors that were thought to be relevant to look at. One thing that is important to note is that much of the ATLAS experiment data has previously used `zlib` with compression level 1 for compression, so it will be treated as something of a baseline and the remaining libraries will be compared to it. Below are tables of libraries that were thought to be particularly notable, for each, we take note of the library used and what compression level, the ratio ((compressed size / original size) * 100), as well as compression and decompression speed.

That's minor, but arguably the text would be easier to read if you put names like zlib, zstd, std::unique_ptr, putPrecompressedData, dataReader etc. in backticks.

[maszyman]

Suggested change

The final step was then to integrate zstd into the system. I had previously moved the logic to write out data into a C++ lambda such that it could more easily be reused. At this step, we have a fragment of data retrieved from our dataReader, I used the simple zstd compression API to take this pre-allocated fragment of data, and then write it out as pre-compressed data with our dataWriter. It is important to use the function putPrecompressedData() rather than putData(), because looking at the internal implementations of the two, we can see that they are wrappers around the same internal function with different settings for whether or not we want to compress the data using zlib. Using putData would result in a file that has been compressed twice which would not be useable in our system. This method of compression is also convenient because it means that individual events remain separate, with all of their metadata being written out as usual. With the decompression step being inserted as the aforementioned fragments are read in by the dataReader, we can decompress each event without having to worry about decompressing and recompressing the full file.

The final step was then to integrate zstd into the system. I had previously moved the logic to write out data into a C++ lambda such that it could more easily be reused. At this step, we have a fragment of data retrieved from our dataReader, I used the simple zstd compression API to take this pre-allocated fragment of data, and then write it out as pre-compressed data with our dataWriter. It is important to use the function putPrecompressedData() rather than putData(), because looking at the internal implementations of the two, we can see that they are wrappers around the same internal function with different settings for whether or not we want to compress the data using zlib. Using putData would result in a file that has been compressed twice which would not be usable in our system. This method of compression is also convenient because it means that individual events remain separate, with all of their metadata being written out as usual. With the decompression step being inserted as the aforementioned fragments are read in by the dataReader, we can decompress each event without having to worry about decompressing and recompressing the full file.

[maszyman]

I know that you defined the ratio in the text, but for people looking only at plots it may be confusing what you mean by compression ratio (recall one of the questions from your presentation at ANL meeting). Perhaps you could add somewhere a clarification that you mean the percentage of the original size?

[maszyman]

I would add here that this project can guide important decision regarding the compression scheme to be implemented in the ATLAS experiment. Also, the fact that ATLAS raw data must be preserved in its original form for an indefinite time. Maybe, I would also stress is that improvement of the compression ratio is a priority, while the speed-up would be a nice advantage.

[maszyman]

You may want to add the references to papers/presentations that I shared with you. Either inline (like here when you write about zlib usage in ATLAS) or collectively at the end.

[maszyman]

You may want to cite RFC document of zstandard and usage in LInux kernal and ROOT framework to support its stability and safety.

[maszyman]

I would add a statement that this study suggests improvements of ~6% in compression ratio at the same CPU cost compared to current method. You may want to add that is a significant as even modest gains can translate into massive savings in storage costs at exabyte scale.

[maszyman]

Please add also acknowledgements to Argonne National Lab :-) (since ANL is participating organization in GSoC, and is the lab of your mentors working on ATLAS@CERN).

[maszyman]

Suggested change

	The final code for the project is available [here](https://github.com/Ish-D/AtlEventProcess)
	The final code for the project is available [here](https://github.com/Ish-D/AtlEventProcess).

[Ish-D]

Updated blog with suggested changes, @maszyman .

[maszyman]

Thanks @Ish-D for the changes. I've left few more (minor) comments. Please have a look.

[maszyman]

Suggested change

All in all, I had a great Summer working with CERN and Argonne National Labto investigate data compression methods for the ATLAS experiment. It was a very good learning experience, although daunting at times, having the opportunity to work in a large existing codebase on a project with real-world impact. I hope that my research over the Summer will prove useful for future compression-related endeavors at the Large Hadron Collider. I would like to extend thanks to my mentor, Maciej Szymanski for this opportunity, as well as all of his very much needed help over the Summer. I definitely could not have completed the project without his guidance and valuable mentorship.

All in all, I had a great Summer working with CERN and Argonne National Lab to investigate data compression methods for the ATLAS experiment. It was a very good learning experience, although daunting at times, having the opportunity to work in a large existing codebase on a project with real-world impact. I hope that my research over the Summer will prove useful for future compression-related endeavors at the Large Hadron Collider. I would like to extend thanks to my mentor, Maciej Szymanski for this opportunity, as well as all of his very much needed help over the Summer. I definitely could not have completed the project without his guidance and valuable mentorship.

[maszyman]

I suggest rephrasing

until it is losslessly compressed at some point in the future

for clarity. The thing is that in science reproducibility is a must. So we need to be able to read this data in future.

[maszyman]

Suggested change

The ideal library would have high compression speed and low compression ratio. We can see that `brotli` covers the widest range, giving us the option to have the slowest compression speed in exchange for the lowest compression ratio, but also the second-fastest speed with the highest ratio. It is also immediately noticeable that `zlib` consistently gives compression ratios that are higher than desirable for the speed being compressed. Both xz and `lzlib` perform very similarly, where they do not offer the same range of options that `brotli` and `zstd` do, and they only operate in the slower and lower compression ratio end of the spectrum, but they are an improvement over `brotli` and `zstd` within that range.

The ideal library would have high compression speed and low compression ratio. We can see that `brotli` covers the widest range, giving us the option to have the slowest compression speed in exchange for the lowest compression ratio, but also the second-fastest speed with the highest ratio. It is also immediately noticeable that `zlib` consistently gives compression ratios that are higher than desirable for the speed being compressed. Both `xz` and `lzlib` perform very similarly, where they do not offer the same range of options that `brotli` and `zstd` do, and they only operate in the slower and lower compression ratio end of the spectrum, but they are an improvement over `brotli` and `zstd` within that range.

[maszyman]

Suggested change

The next step was to go through ATLCopyBSEvent.cpp and modernize it, then integrate `zstd`. I started this process by stripping back the file such that it could only perform the basic operations that it needed: reading, writing, and copying files. Much of the code was removed with the capability to read and write from collections because it was deemed non-essential. Then, starting with this bare-bones version, I ran the program on some existing files that were pre-compressed to various levels, ensuring that it would still be possible to read the files in their varying states and write them out. Once this was ensured, I moved on to modernizing the code. In its old state, there was lots of room for misuse of objects with lots of raw C pointers being passed around, as well as lots of cases of small errors such as passing the raw pointer obtained from std::unique_ptr<T>.get() to a function, rather than moving it. Fixing these issues made the code appear much cleaner, and also meant that we got rid of any potential memory-related issues that could've arisen if the system were used in a larger context. I also added clang-format, to automatically format the code for consistency with the rest of the Atlas codebase.

The next step was to go through `ATLCopyBSEvent`and modernize it, then integrate `zstd`. I started this process by stripping back the file such that it could only perform the basic operations that it needed: reading, writing, and copying files. Much of the code was removed with the capability to read and write from collections because it was deemed non-essential. Then, starting with this bare-bones version, I ran the program on some existing files that were pre-compressed to various levels, ensuring that it would still be possible to read the files in their varying states and write them out. Once this was ensured, I moved on to modernizing the code. In its old state, there was lots of room for misuse of objects with lots of raw C pointers being passed around, as well as lots of cases of small errors such as passing the raw pointer obtained from `std::unique_ptr<T>.get()` to a function, rather than moving it. Fixing these issues made the code appear much cleaner, and also meant that we got rid of any potential memory-related issues that could've arisen if the system were used in a larger context. I also added `clang-format`, to automatically format the code for consistency with the rest of the ATLAS codebase.

[maszyman]

This reported 6% decrease in compression ratio between zstd and zlib at the same CPU cost translates to massive savings when we consider that data is being compressed at exabyte scales.

decrease may not be clear to the reader. I would suggest something along those lines:

The study suggests that zstd offers 6% better compression ratio compared with zlib at the same CPU cost. It should be noted that 6% storage savings is significant at exabyte scale.

[maszyman]

Suggested change

Given the above comparisons, a natural question may be, why ztd? `Brotli` at first glance compresses more efficiently than `zstd`, and since it is the best metric why not choose that? There were a variety of factors that went into this decision: for one, `zstd` performs as a very good all-rounder. It compresses slightly faster than `brotli` and decompresses much faster, sacrificing only 0.5% of the original filesize in compression ratio for this win. Additionally, it offers a fairly wide range of potential settings in the advanced API, including using dictionaries to train better compression and streaming compression for files where we gain the input in fragments at a time. This opens up the possibility of future exploration that I was not able to look at this summer. Finally, `zstd` is guaranteed to be stable. The format used by `zstd` is very stable and well documented, the project is fully open source, it is guaranteed to be supported in the future and sees frequent maintenance/updates, it is already pre-bundled in many popular Linux distributions, and its use in the Linux kernel and CERN ROOT framework are evidence of its ubiquitiousness. For more information about `ztd`'s stability, its [RFC document](https://datatracker.ietf.org/doc/html/rfc8878) provides additional information about how data is compressed, along with security and longevity considerations. The last point was particularly important for CERN, where these files are expected to be compressed and kept for long periods of time till they are decompressed and used at some point in the future, so stability is a must.

Given the above comparisons, a natural question may be, why `zstd`? `Brotli` at first glance compresses more efficiently than `zstd`, and since it is the best metric why not choose that? There were a variety of factors that went into this decision: for one, `zstd` performs as a very good all-rounder. It compresses slightly faster than `brotli` and decompresses much faster, sacrificing only 0.5% of the original file size in compression ratio for this win. Additionally, it offers a fairly wide range of potential settings in the advanced API, including using dictionaries to train better compression and streaming compression for files where we gain the input in fragments at a time. This opens up the possibility of future exploration that I was not able to look at this summer. Finally, `zstd` is guaranteed to be stable. The format used by `zstd` is very stable and well documented, the project is fully open source, it is guaranteed to be supported in the future and sees frequent maintenance/updates, it is already pre-bundled in many popular Linux distributions, and its use in the Linux kernel and CERN ROOT framework are evidence of its ubiquitousness. For more information about `zstd`'s stability, its [RFC document](https://datatracker.ietf.org/doc/html/rfc8878) provides additional information about how data is compressed, along with security and longevity considerations. The last point was particularly important for CERN, where these files are expected to be compressed and kept for long periods of time till they are decompressed and used at some point in the future, so stability is a must.

[Ish-D]

@maszyman updated after feedback from new round of comments.

[maszyman]

Thanks @Ish-D, LGTM!

[maszyman]

Hi @vvolkl 👋

Feel free to review Ishan's blog post, and merge at your convenience, it's all good from my side.

Cheers,
Maciej

[vvolkl]

Very nice report, Thank you!

[maszyman]

Thanks @vvolkl for merging! Is there anything that needs to be done in order this post was published on https://hepsoftwarefoundation.org/gsoc/2024/blogs.html ?

I just noticed that Ishan's markdown document misses this metadata at the beginning:

---
project: 
title: 
author: 
photo: 
date: 
year: 2024
layout: blog_post
logo:
intro: |
---

Is that a problem perhaps?

[hegner]

@maszyman - yes, that is absolutely needed!

[maszyman]

@Ish-D could you please make a PR to create the info needed to publish the blog post?