Converts intermediate connectome building output files for consumption by different programs:
- the binary output of touchdetector to Parquet, to be consumed by functionalizer
- the Parquet output of functionalizer to SONATA, to be read with libsonata
Easiest with the package contained in the BlueBrain fork of Spack.
Clone the project:
$ gh repo clone BlueBrain/parquet-converters -- --recursive --shallow-submodulesThe following dependencies should be provided:
- HDF5
- A MPI implementation, e.g., OpenMPI
- Apache Arrow and Parquet development packages, e.g., following the official instructions
nlohmann_jsonrange-v3- Catch2 (also as submodule)
- CLI11 (also as submodule)
- HighFive (also as submodule)
After ensuring the availability of the dependencies, use the following to build the utilities:
$ cmake -B build -S . -GNinja -DCMAKE_CXX_COMPILER=$(which mpicxx)
$ cmake --build buildSee also the Dockerfile for a full build workflow.
To convert TouchDetector output to Parquet (the output directory is the current directory):
mpirun -np 4 touch2parquet $MY_TD_OUTPUT_DIRECTORY/touchesData.0
This will produce 4 Parquet files, adjust the parallelism accordingly to create more files.
To produce a SONATA file with synapses contained in a population named
All:
mpirun -np 100 parquet2hdf5 $MY_FZ_OUTPUT_DIRECTORY/circuit.parquet edges.h5
All
Creating the synapse index requires a higher parallelism than the initial conversion.
The development of this software was supported by funding to the Blue Brain Project, a research center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss government's ETH Board of the Swiss Federal Institutes of Technology.
Copyright (c) 2017-2024 Blue Brain Project/EPFL