Endianness support

[stukowski]

I have a question about the GSD base format: Can the format handle endianness of different processor architectures? I have not found any hints on this topic in the documentation or in the source code and therefore assume that GSD files are not directly portable between big and little endianness. Is that correct?

We are considering using GSD as the basis for a new standardized file format for the OpenDIS project and defining a new schema for that application. Our hope is that with GSD we could avoid the performance problems of a text-based format like JSON and the dependency problems of a heavy format like HDF5. But the simulation code runs on machines with different endianness. That is why the question of portability is important.

[joaander]

Our hope is that with GSD we could avoid the performance problems of a text-based format like JSON and the dependency problems of a heavy format like HDF5.

These are precisely the reasons why I created GSD. There is also no natural way to change array sizes (i.e. changing number of particles) from one frame to the next in HDF5. The OpenDIS project is welcome to use the GSD format if you find it useful. If the hoomd schema is not sufficient for your needs, you could design your own schema on top of the file layer.

Unfortunately, GSD currently works only with the native machine endian values. The vast majority of systems are little endian so this this has not been a problem so far. GSD could be updated to support reading and writing little endian files on big endian architectures. Would you be willing to put the effort in? gsd is 2.5k lines of C code.

We would need to add byte swaps on writing and reading in many places in the code.

1. The file header (read once on open, written in a handful of locations)
2. The index (used heavily throughout the code and is memory mapped by default on Linux/Mac, see below)
3. The data chunk arrays (trivial to byte swap on read. On write, make the byte swap after the value is added to the write buffer).

The index entries would be the most involved part of the work. GSD uses memory mapped file IO to allow efficient random reads of the index. The gsd_find_chunk API in particular returns a pointer into this mapped memory space and the caller is allowed to read members of this struct. I don't see a way to enable endian swaps in the index entries without breaking this API.

The main optimization that mmap provides is O(1) file open time which is important if you are then going to read only 1 frame out of the file. Do you need this functionality on your big endian systems? There is a GSD_USE_MMAP preprocessor definition that can be disabled (this is normally disabled on Windows) that stores a copy of the index in memory at the cost of O(N_index_entries) time to open the file. If we forced GSD_USE_MMAP off on big endian systems, then the byte swap could be performed in the copy of the index to keep it in the native byte order in RAM.

The write buffer for the index entries will need to be kept native until the last moment before it is written to the file:

gsd/gsd/gsd.c

Lines 2020 to 2022 in 9c68687

	size_t bytes_to_write = sizeof(struct gsd_index_entry) * index_entries_to_write;
	ssize_t bytes_written
	= gsd_io_pwrite_retry(handle->fd, handle->frame_index.data, bytes_to_write, write_pos);

[stukowski]

Thank you for the quick and detailed response to my question. That was more than I expected. I now have a good idea of what steps would be necessary to develop the endian conversion.

I will discuss this information with my partners from the OpenDiS project (who are also reading this) as soon as possible. We still need to define more precisely which use cases are relevant and should determine the design of the file format. I/O performance may not be super critical and we could do without memory mapping on big endian machines. I'll get back to you here. But if you have any questions or comments in the meantime, feel free to contact me privately at my @ovito.org address. Thanks a lot.