blocked CSR for unstructured grids

[angainor]

Often when dealing with problems with many degrees of freedom per node, a blocked CSR is used to represent the compute matrix. In that storage format the grid connectivity itself is represented as with 1 degree of freedom per node (normal graph connectivity matrix), but each vertex is logically responsible for multiple variables, and the sparse matrix contains small, dense blocks of data. Hence, during e.g., sparse matrix vector product it references small contiguous blocks of the input vector.

Exchanging halos for such blocked storage would require a trivial change to GHEX: instead of sending / packing / unpacking a single value from the input vector, GHEX would need to pack / unpack ndof contiguous values. The pointer to the vector position would also need to be trivially scaled by ndof. So effectively GHEX would send/recv [halo_idx*ndof+0, halo_idx*ndof+1, ... halo_idx*ndof+ndof-1] vector subranges.