Issues with I/O

[SBottani]

Hi again Tanguy,

Other annoying issue : I am not able to reload any simulated morphology file in a program for further analysis.

The i/o module documentation is not very helpful and most examples (that do not run) seem outdated (couldn't find any working).

I am updating the doc and docstrings with more help. But need yours to progress !

Several questions

1. "Simulation data" info.json ?

What is in the info.json file generated by dense.io.save_json_info() ? The doc says simulation data. What kind of data ? The generated files seem quite empty. Here is an example from a simulation with 15 neurons. Appear only the names of the kernel parameters, and names of the neurons.

{"kernel": ["max_allowed_resolution", "resolution", "num_mpi_processes", "adaptive_timestep", "record_enabled", "version", "max_synaptic_distance", "environment_initialized", "environment_required", "num_local_threads", "num_virtual_processes", "interactions", "seeds", "time"], "neurons": ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14"]}

I suspect an issue with the generation of this file.

2. swc file loading

From the documentation I understand tthe instruction is.
pop = ds.io.load_swc(swc_folder="my_swc_folder", swc_file="my_swc_file.swc")

It is not very happy

In [6]: pop = ds.io.load_swc(swc_file="2chambers_test.swc")                                                                                                 
import neuron from swc file
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-91c75af034b2> in <module>
----> 1 pop = ds.io.load_swc(swc_file="2chambers_test.swc")

~/.local/lib/python3.8/site-packages/dense-0.1.0-py3.8-linux-x86_64.egg/dense/io/data_swc.py in load_swc(swc_folder, swc_file, info)
     84     """
     85     if swc_file is not None:
---> 86         return Population.from_swc(
     87             NeuronFromSwcFile(swc_file, info))
     88     if swc_folder is not None:

~/.local/lib/python3.8/site-packages/dense-0.1.0-py3.8-linux-x86_64.egg/dense/elements.py in from_swc(cls, population, info)
    859         else:
    860             ensemble = cls(info=population['info'])
--> 861         ensemble._add_swc_population(population['neurons'])
    862         ensemble.sort()
    863         ensemble._idx = {}  # converter from gid to idx

~/.local/lib/python3.8/site-packages/dense-0.1.0-py3.8-linux-x86_64.egg/dense/elements.py in _add_swc_population(self, neurons)
    996             axon, dendrites = _get_swc_struct(neuron=neurons[neuron]['data'])
    997             try:
--> 998                 position = self.info["neurons"][str(
    999                     neurons[neuron]['gid'])]['position']
   1000             except KeyError:

TypeError: 'NoneType' object is not subscriptable

And if I put the swc file in a folder, then we get

In [7]: pop = ds.io.load_swc(swc_folder="2culture_swc",swc_file="2chambers_test.swc")                                                                       
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-7-62cd3b497c6a> in <module>
----> 1 pop = ds.io.load_swc(swc_folder="2culture_swc",swc_file="2chambers_test.swc")

~/.local/lib/python3.8/site-packages/dense-0.1.0-py3.8-linux-x86_64.egg/dense/io/data_swc.py in load_swc(swc_folder, swc_file, info)
     84     """
     85     if swc_file is not None:
---> 86         return Population.from_swc(
     87             NeuronFromSwcFile(swc_file, info))
     88     if swc_folder is not None:

~/.local/lib/python3.8/site-packages/dense-0.1.0-py3.8-linux-x86_64.egg/dense/elements.py in from_swc(cls, population, info)
    859         else:
    860             ensemble = cls(info=population['info'])
--> 861         ensemble._add_swc_population(population['neurons'])
    862         ensemble.sort()
    863         ensemble._idx = {}  # converter from gid to idx

~/.local/lib/python3.8/site-packages/dense-0.1.0-py3.8-linux-x86_64.egg/dense/elements.py in _add_swc_population(self, neurons)
    994         for neuron in neurons:
    995             gid = neurons[neuron]['gid']
--> 996             axon, dendrites = _get_swc_struct(neuron=neurons[neuron]['data'])
    997             try:
    998                 position = self.info["neurons"][str(

~/.local/lib/python3.8/site-packages/dense-0.1.0-py3.8-linux-x86_64.egg/dense/io/data_swc.py in GetSWCStructure(neuron, plot, neuron_gid, axon_ID, dendrite_ID)
    293 
    294     else:
--> 295         import btmorph2
    296         if isinstance(neuron, btmorph2.NeuronMorphology):
    297             if plot:

ModuleNotFoundError: No module named 'btmorph2'

(Alas btmorph2 is in Python2 and I could install it to check what happened with it)

Alternatively, some examples show to import a morphology the following coed:

from dense.elements import Population
pop = Population.from_swc("my_swc_file.swc")

But if I'm not wrong this is deprecated syntax, and anyway it does not work.

pop = Population.from_swc("2chambers_test.swc")                                                                                                     
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-3e1b26e8106c> in <module>
----> 1 pop = Population.from_swc("2chambers_test.swc")

~/.local/lib/python3.8/site-packages/dense-0.1.0-py3.8-linux-x86_64.egg/dense/elements.py in from_swc(cls, population, info)
    858             ensemble = cls(info=population['info'])
    859         else:
--> 860             ensemble = cls(info=population['info'])
    861         ensemble._add_swc_population(population['neurons'])
    862         ensemble.sort()

TypeError: string indices must be integers

I'm quite at a loss here. Thank you in advance
Samuel

[Silmathoron]

Hi Samuel,
unfortunately this part was done by @cocconat and I have little knowledge about it... still, I will try to have a look on Monday.
I'm pretty sure the .json file contains exactly what it's supposed to, though (his idea was to save the configuration of the run).

Anyway, regarding SWC load, I'm not sure we have much... and anything containing btmorph is way past expired... I'll try to figure it out.
From what I remember, the initial idea was that we would use other libraries to analyze results from DeNSE, which is why this is currently not our strongsuit (the O part of I/O should be OK, but the I is probably lacking).

[SBottani]

OK, so at least I understand from your answer what was the purpose of the json file.
I'll update the doc to explain that it deals to save the simulation configuration.
This would be indeed useful. Yet I fear it is not working. On the examples I tried, all configuration parameter values were empty. Only parameter names being there.
I'll have a look further into it.

For the Input... I supposed from your beautifull thesis images the analysis part was working well. If we can't rely anymore on it and btmorph we should find something else.

Let us speak about this another time then.

[Silmathoron]

@SBottani with #36 being merged, could you update the status of this issue?
Are there still things that do not work?

More importantly, what do we want to achieve on the import side and what would you say is still missing?

[SBottani]

Still need to work on it sorry.

The json file seems still empty.

Necessary:

• to be sure the exported swc and/or nml files can indeed be opened and processed by other visualizations apps

[SBottani]

Desired I/O feature:

• importing swc or nmn morphology file of a collection of neurons to be able to generate the interaction graph.

[Silmathoron]

@SBottani for .swc, isn't that what Population.from_swc is supposed to do? Could you give some details about what this currently gives?

For .nml is think we don't have anything at the moment.

[SBottani]

This is what I wanted to test when falling on issue #41 ( wanted to run the circular.py to generate a morphology in a circular environment and then test loading the swc file). Last time I attempted using Population.from_swc I did not succed.

[SBottani]

I think there's an issue in /dense/io/data_swc.py and a dependecy to btmorph2. I'll try to look into it.