Skip to content

Commit

Permalink
updates to the tools and the tests
Browse files Browse the repository at this point in the history
  • Loading branch information
@jofrony
jofrony committed Jul 24, 2023
1 parent 240c8d6 commit eea6f14
Show file tree
Hide file tree
Showing 23 changed files with 57,317 additions and 94 deletions.
1 change: 1 addition & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Extra transfer files for model prep
tools/test_data/
Planert2010/figures/
tmp/

# Byte-compiled / optimized / DLL files
__pycache__/
Expand Down
138 changes: 138 additions & 0 deletions examples/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
# How to transfer your model from BluePyOpt to BasalGangliaData (using Snudda format)

# BluePyOpt

To create single multi-compartmental models, the lab utilizes [BluePyOpt](https://github.com/BlueBrain/BluePyOpt). The optimization is started using a collection of code* with the "set up" of the
optimization is done with the following code:

```
optimiser = bpopt.optimisations.DEAPOptimisation(
evaluator=evaluator,
offspring_size=offspring_size,
map_function=lview.map_sync,
seed=1)
pop, hof, log, hist = optimiser.run(max_ngen=ngenerations)
```

see the method "run" in BluePyOpt [here](https://github.com/BlueBrain/BluePyOpt/blob/dfd202904c4f497c54574c7f321a95bb5183438b/bluepyopt/deapext/optimisations.py#L253)

The "pop","hof","log" and "hist" are returned from "run". The "hof" contains the "best" parameter sets of the optimization. The "hof" is saved into a json-file by the following code:

```
import json
best_models = []
for record in hof:
params = evaluator.param_dict(record)
best_models.append(params)
with open('best_models.json', 'w') as fp:
json.dump(best_models, fp, indent=4)
```
The file can be further filtered by different validation scripts* but the structure should be the same as in 'best_models.json' (a list of dictionaries). See examples/ for examples of each file required in the conversion from BluePyOpt to Snudda.

Some people in the lab use 'best_models.json', but further validation following the optimization should filter this list and these models are saved in 'hall_of_fame.json'. This file has the same structure but might contain fewer models, as other features are measured during validation compared to optimization.

If the optimization is also tested on several different morphologies, a third file called 'val_models.json' is required (See examples/ for an example of 'val_model.json').


*for more information on the optimization contact Alex Kozlov or Ilaria Carannante.

## Option 1: Give the path to specific files

### Required files
Mandatory
* parameters.json
* mechanisms.json
VERSION - either 1 or 2 or 2 + 3
1. best_models.json (if you are using the direct output of the optimizer)
2. hall_of_fame.json (if you have filtered the parameter sets against more validations)
3. val_models.json (if you have varied the morphology used within the original optimization and hence have more morph-parameter combinations)
The model optimisation could be a folder, containing the follow files and subdirectories:

model/
config/
parameters.json
mechanisms.json
morphology/
contain one or several morphologies (.swc)
used for the model
hall_of_fame.json ( contain the parameter sets - the results of the optimisation)
val_models.json ( optional file, if several morphologies are used, the parameter sets which match each morphology)
For an example of the structure and contents of the files, see **BasalGangliaData/tests/test_data/example_variation_source**

*Contact Alex Kozlov for more information

# The steps

### Create your own notebook and copy-paste the code to perform each step individually or utilize the class TransferBluePyOptToSnudda.

The transfer has been divided into several steps.

Create directory for the model.
```
Within BasalGangliaData, the models used in Snudda are saved under
BasalGangliaData/data/neurons/name_of_nucleus
**If the nucleus does not exist, add a folder for the new nucleus**
Next create (if it does not already exist), a folder for each cell type within the nucleus
Lastly, create the folder for each model of the cell type
(this folder will be the **destination** used in the code below)
For example,
BasalGanglia/data/neurons/newnucleus/new_celltype/new_model
```

### Add tools to your path

```
import sys
sys.path.append("../tools")
source = "where the Bluepyopt optimisation, with the structure described above"
destination = "BasalGanglia/data/neurons/newnucleus/new_celltype/new_model"
```

### Transfer mechanisms

```
from transfer.mechanisms import transfer_mechanisms
transfer_mechanisms(source=source, destination=destination)
```

### Transfer parameters

```
from transfer.parameters import transfer_parameters
transfer_parameters(source=source,
destination=destination,
selected=True)
```

### Transfer selected models from val_models.json


```
from transfer.selected_models import transfer_selected_models
transfer_selected_models(source=source, destination=destination)
```

### Transfer morphologies

```
from transfer.morphology import transfer_morphologies
transfer_morphologies(source=source,
destination=destination,
selected=True)
```

### Create the meta.json which combines all information on the model

```
from meta.create_meta import write_meta
write_meta(directory=destination, selected=True)
```
232 changes: 232 additions & 0 deletions examples/example-option-2/best_models.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,232 @@
[
{
"g_pas.axonal": 0.000289764140870004,
"g_pas.basal": 5.898654312745068e-05,
"g_pas.somatic": 0.0005424674497187078,
"gbar_naf_ms.axonal": 1.912987850428717,
"gbar_kas_ms.axonal": 0.0005986203683081819,
"gbar_Im_ms.axonal": 0.048854852118683234,
"gbar_naf_ms.somatic": 19.67872706705603,
"gbar_kaf_ms.somatic": 1.6455236658644132,
"gbar_kas_ms.somatic": 1.5522131053164893e-07,
"gbar_kdr_ms.somatic": 0.0009767044304803505,
"gbar_kir_ms.somatic": 0.0027806790825306567,
"gbar_bk_ms.somatic": 1.4757260390474007e-05,
"gbar_sk_ms.somatic": 9.24153705842713e-05,
"gbar_Im_ms.somatic": 0.00038621296183501797,
"gbar_naf_ms.basal": 0.003307569668585907,
"gbar_kaf_ms.basal": 0.010565081414299675,
"gbar_kas_ms.basal": 1.3806474267247748e-05,
"gbar_kdr_ms.basal": 0.00017387376352705973,
"gbar_kir_ms.basal": 0.00013832919173207862,
"gbar_bk_ms.basal": 2.6969759458566956e-05,
"gbar_sk_ms.basal": 4.4733433938030425e-06
},
{
"g_pas.axonal": 0.000289764140870004,
"g_pas.basal": 5.898654312745068e-05,
"g_pas.somatic": 0.0005608913056047753,
"gbar_naf_ms.axonal": 2.2356202678732204,
"gbar_kas_ms.axonal": 0.012982595405314516,
"gbar_Im_ms.axonal": 0.0444021348894177,
"gbar_naf_ms.somatic": 19.67872706705603,
"gbar_kaf_ms.somatic": 1.4898348311238556,
"gbar_kas_ms.somatic": 3.53394125317253e-05,
"gbar_kdr_ms.somatic": 0.0009767044304803505,
"gbar_kir_ms.somatic": 0.0027806790825306567,
"gbar_bk_ms.somatic": 2.000010322027415e-05,
"gbar_sk_ms.somatic": 9.633220545562533e-05,
"gbar_Im_ms.somatic": 0.00038621296183501797,
"gbar_naf_ms.basal": 0.0026486226606414185,
"gbar_kaf_ms.basal": 0.015442481265960305,
"gbar_kas_ms.basal": 4.115713069723965e-05,
"gbar_kdr_ms.basal": 0.00017387376352705973,
"gbar_kir_ms.basal": 0.00013832919173207862,
"gbar_bk_ms.basal": 2.845173075172165e-05,
"gbar_sk_ms.basal": 4.4733433938030425e-06
},
{
"g_pas.axonal": 0.000372735228283294,
"g_pas.basal": 5.898654312745068e-05,
"g_pas.somatic": 0.0006448158564273618,
"gbar_naf_ms.axonal": 2.7615251806523116,
"gbar_kas_ms.axonal": 0.012982595405314516,
"gbar_Im_ms.axonal": 0.02617431866250017,
"gbar_naf_ms.somatic": 19.522989061507893,
"gbar_kaf_ms.somatic": 1.4898348311238556,
"gbar_kas_ms.somatic": 3.53394125317253e-05,
"gbar_kdr_ms.somatic": 0.0009997488969778004,
"gbar_kir_ms.somatic": 0.0027806790825306567,
"gbar_bk_ms.somatic": 5.5525458027222725e-05,
"gbar_sk_ms.somatic": 9.633220545562533e-05,
"gbar_Im_ms.somatic": 0.00034429372050213426,
"gbar_naf_ms.basal": 0.0026486226606414185,
"gbar_kaf_ms.basal": 0.015442481265960305,
"gbar_kas_ms.basal": 4.137841591156171e-05,
"gbar_kdr_ms.basal": 2.0613714748991303e-05,
"gbar_kir_ms.basal": 0.000138848764124438,
"gbar_bk_ms.basal": 2.845173075172165e-05,
"gbar_sk_ms.basal": 1.5475972608212357e-06
},
{
"g_pas.axonal": 0.000289764140870004,
"g_pas.basal": 5.898654312745068e-05,
"g_pas.somatic": 0.0005608913056047753,
"gbar_naf_ms.axonal": 2.0034553639881008,
"gbar_kas_ms.axonal": 0.012982595405314516,
"gbar_Im_ms.axonal": 0.038361495398255056,
"gbar_naf_ms.somatic": 19.94846967849026,
"gbar_kaf_ms.somatic": 1.4898348311238556,
"gbar_kas_ms.somatic": 0.00301476657801522,
"gbar_kdr_ms.somatic": 0.0009767044304803505,
"gbar_kir_ms.somatic": 0.0027806790825306567,
"gbar_bk_ms.somatic": 2.000010322027415e-05,
"gbar_sk_ms.somatic": 9.633220545562533e-05,
"gbar_Im_ms.somatic": 0.00038621296183501797,
"gbar_naf_ms.basal": 0.01154937260461457,
"gbar_kaf_ms.basal": 0.013774210001463656,
"gbar_kas_ms.basal": 4.115713069723965e-05,
"gbar_kdr_ms.basal": 0.00017387376352705973,
"gbar_kir_ms.basal": 0.00013832919173207862,
"gbar_bk_ms.basal": 2.845173075172165e-05,
"gbar_sk_ms.basal": 4.4733433938030425e-06
},
{
"g_pas.axonal": 0.00046134058400490447,
"g_pas.basal": 6.765120536453554e-05,
"g_pas.somatic": 0.00031922573198067683,
"gbar_naf_ms.axonal": 1.1740520799265424,
"gbar_kas_ms.axonal": 0.012982595405314516,
"gbar_Im_ms.axonal": 0.05128406902658957,
"gbar_naf_ms.somatic": 19.653306198239253,
"gbar_kaf_ms.somatic": 1.4898348311238556,
"gbar_kas_ms.somatic": 3.53394125317253e-05,
"gbar_kdr_ms.somatic": 0.0007258683462697301,
"gbar_kir_ms.somatic": 0.0023953091441674687,
"gbar_bk_ms.somatic": 2.000010322027415e-05,
"gbar_sk_ms.somatic": 9.633220545562533e-05,
"gbar_Im_ms.somatic": 0.0002548809269820513,
"gbar_naf_ms.basal": 0.01898304074450227,
"gbar_kaf_ms.basal": 0.011383013127619603,
"gbar_kas_ms.basal": 4.115713069723965e-05,
"gbar_kdr_ms.basal": 0.00017387376352705973,
"gbar_kir_ms.basal": 0.00017889174247442567,
"gbar_bk_ms.basal": 1.0373579107052646e-06,
"gbar_sk_ms.basal": 7.324763801809509e-06
},
{
"g_pas.axonal": 0.0003544105073629326,
"g_pas.basal": 5.898654312745068e-05,
"g_pas.somatic": 0.0005608913056047753,
"gbar_naf_ms.axonal": 4.526489430493155,
"gbar_kas_ms.axonal": 0.012727006840461467,
"gbar_Im_ms.axonal": 0.07938298238575137,
"gbar_naf_ms.somatic": 19.430264553232107,
"gbar_kaf_ms.somatic": 1.4898348311238556,
"gbar_kas_ms.somatic": 2.497717350876012e-05,
"gbar_kdr_ms.somatic": 0.0009767044304803505,
"gbar_kir_ms.somatic": 0.002908879471585272,
"gbar_bk_ms.somatic": 2.0653250882227896e-05,
"gbar_sk_ms.somatic": 9.633220545562533e-05,
"gbar_Im_ms.somatic": 0.00035286756961802975,
"gbar_naf_ms.basal": 0.01118238525229902,
"gbar_kaf_ms.basal": 0.010162983834883852,
"gbar_kas_ms.basal": 4.115713069723965e-05,
"gbar_kdr_ms.basal": 0.0003539638004493841,
"gbar_kir_ms.basal": 0.00014060180564900423,
"gbar_bk_ms.basal": 3.286006308208744e-05,
"gbar_sk_ms.basal": 4.225887383787686e-06
},
{
"g_pas.axonal": 0.000289764140870004,
"g_pas.basal": 5.898654312745068e-05,
"g_pas.somatic": 0.0005608913056047753,
"gbar_naf_ms.axonal": 2.2356202678732204,
"gbar_kas_ms.axonal": 0.010501968904469841,
"gbar_Im_ms.axonal": 0.0444021348894177,
"gbar_naf_ms.somatic": 18.89927867926048,
"gbar_kaf_ms.somatic": 1.4898348311238556,
"gbar_kas_ms.somatic": 3.53394125317253e-05,
"gbar_kdr_ms.somatic": 0.0009995777305256799,
"gbar_kir_ms.somatic": 0.0027806790825306567,
"gbar_bk_ms.somatic": 2.000010322027415e-05,
"gbar_sk_ms.somatic": 9.647445889849785e-05,
"gbar_Im_ms.somatic": 0.0009827373227784268,
"gbar_naf_ms.basal": 0.0026486226606414185,
"gbar_kaf_ms.basal": 0.014552856359892926,
"gbar_kas_ms.basal": 4.115713069723965e-05,
"gbar_kdr_ms.basal": 0.00014697837436541316,
"gbar_kir_ms.basal": 0.00013040401612049882,
"gbar_bk_ms.basal": 2.845173075172165e-05,
"gbar_sk_ms.basal": 3.0649089613900916e-06
},
{
"g_pas.axonal": 0.000464509985427581,
"g_pas.basal": 6.648854388610662e-05,
"g_pas.somatic": 0.0005608913056047753,
"gbar_naf_ms.axonal": 2.7146644720672723,
"gbar_kas_ms.axonal": 0.012982595405314516,
"gbar_Im_ms.axonal": 0.053678096741935924,
"gbar_naf_ms.somatic": 19.67872706705603,
"gbar_kaf_ms.somatic": 1.4898348311238556,
"gbar_kas_ms.somatic": 3.53394125317253e-05,
"gbar_kdr_ms.somatic": 0.0009767044304803505,
"gbar_kir_ms.somatic": 0.0027806790825306567,
"gbar_bk_ms.somatic": 2.000010322027415e-05,
"gbar_sk_ms.somatic": 9.633220545562533e-05,
"gbar_Im_ms.somatic": 0.00038621296183501797,
"gbar_naf_ms.basal": 0.0026486226606414185,
"gbar_kaf_ms.basal": 0.015442481265960305,
"gbar_kas_ms.basal": 4.115713069723965e-05,
"gbar_kdr_ms.basal": 1.2553027325013942e-05,
"gbar_kir_ms.basal": 0.00013832919173207862,
"gbar_bk_ms.basal": 2.845173075172165e-05,
"gbar_sk_ms.basal": 5.440159907845509e-06
},
{
"g_pas.axonal": 0.000289764140870004,
"g_pas.basal": 5.898654312745068e-05,
"g_pas.somatic": 0.0006540561504876143,
"gbar_naf_ms.axonal": 1.9991807502095007,
"gbar_kas_ms.axonal": 0.011765151763443737,
"gbar_Im_ms.axonal": 0.006504565769292422,
"gbar_naf_ms.somatic": 18.980195228260783,
"gbar_kaf_ms.somatic": 1.6176224883717782,
"gbar_kas_ms.somatic": 4.088269360091441e-06,
"gbar_kdr_ms.somatic": 0.0009100666818358603,
"gbar_kir_ms.somatic": 0.0027570164719982896,
"gbar_bk_ms.somatic": 8.542937910939434e-06,
"gbar_sk_ms.somatic": 9.924883651864302e-05,
"gbar_Im_ms.somatic": 0.0003839017065527756,
"gbar_naf_ms.basal": 0.004787170559633562,
"gbar_kaf_ms.basal": 0.015442481265960305,
"gbar_kas_ms.basal": 4.121395918890036e-05,
"gbar_kdr_ms.basal": 0.00018248300040972187,
"gbar_kir_ms.basal": 0.00012839014644958288,
"gbar_bk_ms.basal": 2.845173075172165e-05,
"gbar_sk_ms.basal": 4.620352966524964e-06
},
{
"g_pas.axonal": 0.000289764140870004,
"g_pas.basal": 5.4075794421199945e-05,
"g_pas.somatic": 0.0005608913056047753,
"gbar_naf_ms.axonal": 2.2356202678732204,
"gbar_kas_ms.axonal": 0.00015237514137519947,
"gbar_Im_ms.axonal": 0.0444021348894177,
"gbar_naf_ms.somatic": 19.598830054765703,
"gbar_kaf_ms.somatic": 1.568176631815741,
"gbar_kas_ms.somatic": 0.005403895093902649,
"gbar_kdr_ms.somatic": 0.0009767044304803505,
"gbar_kir_ms.somatic": 0.0027806790825306567,
"gbar_bk_ms.somatic": 2.000010322027415e-05,
"gbar_sk_ms.somatic": 9.633220545562533e-05,
"gbar_Im_ms.somatic": 0.0011852621591995514,
"gbar_naf_ms.basal": 0.011887232891004315,
"gbar_kaf_ms.basal": 0.011951975583603234,
"gbar_kas_ms.basal": 2.494822677589877e-06,
"gbar_kdr_ms.basal": 0.00019008299126809595,
"gbar_kir_ms.basal": 0.00013832919173207862,
"gbar_bk_ms.basal": 1.6353380990540888e-05,
"gbar_sk_ms.basal": 6.709483323830171e-06
}
]
Loading

0 comments on commit eea6f14

Please sign in to comment.