ITStrF01
Here is an explanation of the ITStrF01 fileformat used by iontof in the following files:
- .ita
- .itm
- .its
- .shi
- .isi
- .spm
Note that this was done by pure reverse-engineering and that this documentation might not be 100% accurate with reality. Nevertheless this procedure seems to work well with a huge amount of data already treated and parsed by the developer.
The fileformat ITStrF01 is composed by 8-bytes forming the word "ITStrF01" followed by an unknown number of blocks of unknown size. The size can be found in the header of each block. The blocks form a filesystem type where the first block is "root" and its value is a list of children's block. The leaf-blocks are of type 00 19 00 00 00 or 80 19 00 00 00 and contains data.
Each block is composed by a header followed by its name and then its value/data. All values are in uint32.
N: gives the number of children of block types 01/03.
ID: If several blocks have the same name, then they should have different ID. It usually starts at 0 and increase monotonically for all following blocks. It was found that block which are overwritten (e.g. information about the mass calibration sf/k0) are increased, but the blocks of lower ID are not kept. In order to solve this issue pySPM.Block.goto provides the lazy argument, which can be set to True in order to return the first children found of a given name without having to know its ID.
Only 4 different types have been encountered in ITA/ITM files. Those are:
Those block have no children and the value contains only data. The data type (string/int/float/etc.) is not written in the header. The user should know what is the type of data depending on the block path. Sometime key/values are stored.
Those block are block containing children. Similar to a folder. The value is then composed only of headers defining the name and position of all the children (see Value header and Child header below).
Big block are chunked in several smaller blocks. Type 01 are chunks in several blocks and all "Next-Block" are of type 03. The structure is exactly the same as 01 type though.
Those blocks have no children and contain data compressed with zlib. It usually stores array of data as a stack of floats.
Blocks of type 01 and 03 have children. There value thus start with a header composed of 41 bytes. The values a is equivalent to the block name, b to Block ID, c to N, d to length1, e is found to be equal to 0.
The 41 bytes of the value header is then followed by several 33-bytes long block giving info for every children. All values are in uint32 or uint64 if not explicitly specified.
S: is usually 00 or 01 and its meaning is unclear
N is usually the number of children. In the past it was thought that some block have children even when N=0 and that nums gives the exact number of children in this special case. We believe now that this was erroneous and that when N=0, no children are in the block. Now the block size is fixed once written. For this purpose iontof write specially long block header even without children so that the block can be populated afterward without resizing the block (which would be too time consuming). We now believe that nums is an indicator on the reserved size. So if nums=20, it means that the block can be populated by 20 children. If this is the case, that would be strange as the header depends on the block_name of the children. In any case we don't need this information (nums) in order to read the data properly. Note: If a children is "erased", usually N is brought back to 0 without "erasing" the information of the block itself.
The blocks after "nums" (green+a+b+c+d+e) are all equal to 0 if there is no Next Block. Otherwise it gives information about NextBlock. It's why the "green" block can only have value of 00 00 00 00 00 or 03 19 00 00 00. c will tell how many children are present in the Next-Block. d seems to be in the same range as the size, but not exactly equal to it. We don't use those information as it's redundant and those can be read in the header of the Next-Block.