Release v0.4.1 (Merge pull request #212 from ICB-DCM/develop)

Release v0.4.1

* Add Zenodo badge (Closes #189)
* Update AMICI and disentangle parallel AMICI output during optimization and simulation (#204)
* Autodeploy to dockerhub using github action (#205)
* Improvements minibatch (#206) (https://www.biorxiv.org/content/10.1101/859884v1)
* implemented changes from supermuc with fixes in minibatch rescue interceptor and line search algorithm
* Use PEtab parameter prior for starting point sampling (#194) 

Minor:
* Explicitly enable oversubscription since we are using OpenMPI
* Disentangle parallel AMICI output during optimization (Closes #202)
* Update list of references
* Fix model naming
* Fix docker MPI issues 
  * due to lack of SYS_PTRACE permissions (disable Vader) (Closes #185)
  * Disable Infiniband for OpenMPI
  * enable tcp via loopback
* Set RelWithDebInfo as default build type
* Update AMICI to v0.10.16 and PEtab v0.0.0a17 (#209)
* Disable saving states and observables during optimization, as the current format does not support varying number of timepoints and crashes
* Allow creation of virtual environment in the absence of ensurepip
* Fix uninitialized usage warnings

.github/workflows/deploy_dockerhub.yml

@@ -0,0 +1,18 @@
+# https://github.com/marketplace/actions/publish-docker
+name: Deploy to dockerhub
+on: [push]
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@master
+    - run: git archive -v -o container/charliecloud/parpe_base/parpe.tar.gz --format=tar.gz HEAD
+    - name: Publish to Registry
+      uses: elgohr/Publish-Docker-Github-Action@2.8
+      with:
+        name: dweindl/parpe
+        username: ${{ secrets.DOCKER_USERNAME }}
+        password: ${{ secrets.DOCKER_PASSWORD }}
+        workdir: container/charliecloud/parpe_base/
+        dockerfile: Dockerfile
+        tag_names: true

CMakeLists.txt

@@ -1,5 +1,5 @@
-cmake_minimum_required(VERSION 3.6)
-cmake_policy(VERSION 3.6)
+cmake_minimum_required(VERSION 3.7)
+cmake_policy(VERSION 3.7)
 
 project(parpe)
 

CMakeModules/BuildType.cmake

@@ -1,6 +1,6 @@
 # Set a default build type if none was specified.
 # If inside git repository build debug, otherwise release.
-set(default_build_type "Release")
+set(default_build_type "RelWithDebInfo")
 if(EXISTS "${CMAKE_SOURCE_DIR}/.git")
   set(default_build_type "Debug")
 endif()

README.md

@@ -1,6 +1,8 @@
 [![Run Status](https://api.shippable.com/projects/59463d3e8993d7070010407b/badge?branch=master)](https://app.shippable.com/github/dweindl/parPE)
 [![Coverage Badge](https://api.shippable.com/projects/59463d3e8993d7070010407b/coverageBadge?branch=master)](https://app.shippable.com/github/dweindl/parPE)
 [![Codacy Badge](https://api.codacy.com/project/badge/Grade/1f1ee5a0d90d431499f200a148fb7fdc)](https://www.codacy.com?utm_source=github.com&utm_medium=referral&utm_content=ICB-DCM/parPE&utm_campaign=Badge_Grade)
+[![DOI](https://zenodo.org/badge/92953596.svg)](https://zenodo.org/badge/latestdoi/92953596)
+
 # parPE
 
 The *parPE* library provides functionality for solving large-scale parameter
@@ -110,6 +112,12 @@ parPE is being used or has been used in the following projects:
   Bioinformatics, btz581, [doi:10.1093/bioinformatics/btz581](https://doi.org/10.1093/bioinformatics/btz581)
   (preprint: [doi:10.1101/579045](https://www.biorxiv.org/content/10.1101/579045v1)).
 
+- Paul Stapor, Leonard Schmiester, Christoph Wierling, Bodo Lange,
+  Daniel Weindl, and Jan Hasenauer. 2019.
+  *Mini-Batch Optimization Enables Training of Ode Models on Large-Scale Datasets.*
+  bioRxiv. Cold Spring Harbor Laboratory.
+  preprint: [doi:10.1101/859884](https://doi.org/10.1101/859884).
+
 - [CanPathPro](http://canpathpro.eu/)
 
 

benchmark_collection/import_and_run.sh

@@ -54,7 +54,7 @@ CMD="${PARPE_DIR}/misc/generateHDF5DataFileFromText.py \
     -d ${AMICI_MODEL_DIR} \
     -m ${PETAB_MODEL_DIR}/measurementData_${MODEL_NAME}.tsv \
     -c ${PETAB_MODEL_DIR}/experimentalCondition_${MODEL_NAME}.tsv \
-    -n model_${MODEL_NAME} \
+    -n ${MODEL_NAME} \
     -p ${PETAB_MODEL_DIR}/parameters_${MODEL_NAME}.tsv"
 echo $CMD
 $CMD

container/charliecloud/parpe_base/install.sh

@@ -96,19 +96,20 @@ apt-get install -q -y git
 #echo "================= Adding awscli 1.14.64 ============"
 #sudo pip install -q 'awscli==1.14.64'
 
-#echo "================= parPE requirements ============"
-apt-get install gfortran libmpich-dev libatlas-base-dev libboost-all-dev libhdf5-dev cmake libceres-dev coinor-libipopt-dev swig3.0 python3-venv hdf5-tools libpython-dev
+echo "================= parPE requirements ============"
+# using openmpi coming with libboost-all-dev instead of libmpich-dev
+apt-get install gfortran libatlas-base-dev libboost-all-dev libhdf5-dev cmake libceres-dev coinor-libipopt-dev swig3.0 python3-venv hdf5-tools libpython-dev
 # for setuptools to find:
 ln -s /usr/bin/swig3.0 /usr/bin/swig
 python3 -m pip install --upgrade pip
 pip3 install -U setuptools pkgconfig wheel
 
 # echo "================= Intalling Shippable CLIs ================="
-# 
+#
 # git clone https://github.com/Shippable/node.git nodeRepo
 # ./nodeRepo/shipctl/x86_64/Ubuntu_16.04/install.sh
 # rm -rf nodeRepo
-# 
+#
 # echo "Installed Shippable CLIs successfully"
 # echo "-------------------------------------"
 

container/charliecloud/parpe_base/install_parpe.sh

@@ -3,10 +3,10 @@ set -e
 
 cd
 
-# Clone clean repository
-git clone --single-branch --branch feature_charlie --depth=1 https://github.com/ICB-DCM/parPE.git
+# unpack git archive
+mkdir parPE && cd parPE
+tar -xzf /u18/parpe.tar.gz
 
-cd parPE
 export PARPE_BASE=$(pwd)
 
 # Build dependencies
@@ -36,6 +36,7 @@ CC=mpicc CXX=mpiCC cmake \
       -DCERES_INCLUDE_DIRS="/usr/include/;/usr/include/eigen3" \
       -DMPI_INCLUDE_DIRS=/usr/include/openmpi-x86_64/ \
       -DBUILD_TESTS=ON \
+      "-DTESTS_MPIEXEC_COMMAND=mpiexec;--allow-run-as-root;-n;4;--oversubscribe;--mca;btl_vader_single_copy_mechanism;none;--mca;btl;^openib;--mca;oob_tcp_if_include;lo;--mca;btl_tcp_if_include;lo;--mca;orte_base_help_aggregate;0" \
       ..
 make -j12 VERBOSE=1
 

deps/AMICI/.github/workflows/sbml-semantic-test-suite.yml

@@ -4,7 +4,6 @@ on:
     branches:
       - develop
       - master
-      - test_action
   pull_request:
     branches:
       - master

deps/AMICI/.github/workflows/test-large-model.yml

@@ -0,0 +1,82 @@
+name: Performance test
+on:
+  push:
+    branches:
+      - develop
+      - master
+      - feature_sparse_quadratures
+
+  pull_request:
+    branches:
+      - master
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v1
+      with:
+        fetch-depth: 20
+
+    # install dependencies
+    - name: apt
+      run: |
+        sudo apt-get install -y swig3.0 libatlas-base-dev \
+          && sudo ln -s /usr/bin/swig3.0 /usr/bin/swig
+    - name: pip
+      run: |
+        pip3 install --upgrade --user wheel \
+          && pip3 install --upgrade --user setuptools
+    - run: pip3 install petab shyaml
+    - run: |
+        echo ::add-path::${HOME}/.local/bin/
+        echo ::add-path::${GITHUB_WORKSPACE}/tests/performance/
+    # install AMICI
+    - name: Create AMICI sdist
+      run: |
+        cd python/sdist \
+          && check_time.sh create_sdist /usr/bin/python3 setup.py sdist
+    - name: Install AMICI sdist
+      run: |
+        AMICI_PARALLEL_COMPILE=2 check_time.sh \
+            install_sdist pip3 install -v --user \
+            $(ls -t python/sdist/dist/amici-*.tar.gz | head -1)
+
+    # retrieve test model
+    - run: git clone --depth 1 https://github.com/ICB-DCM/CS_Signalling_ERBB_RAS_AKT
+
+    # import test model
+    - name: Import test model
+      run: |
+        cd CS_Signalling_ERBB_RAS_AKT \
+          && check_time.sh \
+          petab_import amici_import_petab.py -v \
+            -s 'PEtab/CS_Signalling_ERBB_RAS_AKT_petab.xml' \
+            -c 'PEtab/conditions_petab.tsv' \
+            -m 'PEtab/measurements_petab.tsv' \
+            -p 'PEtab/parameters_petab.tsv' --no-compile
+
+    # install model package
+    - name: Install test model
+      run: |
+        check_time.sh install_model \
+          python3 CS_Signalling_ERBB_RAS_AKT/CS_Signalling_ERBB_RAS_AKT_petab/setup.py install --user
+
+    # run simulations
+    - name: forward_simulation
+      run: |
+        check_time.sh forward_simulation tests/performance/test.py forward_simulation
+    - name: forward_sensitivities
+      run: |
+        check_time.sh forward_sensitivities tests/performance/test.py forward_sensitivities
+    - name: adjoint_sensitivities
+      run: |
+        check_time.sh adjoint_sensitivities tests/performance/test.py adjoint_sensitivities
+    - name: forward_simulation_non_optimal_parameters
+      run: |
+        check_time.sh forward_simulation_non_optimal_parameters tests/performance/test.py forward_simulation_non_optimal_parameters
+    - name: adjoint_sensitivities_non_optimal_parameters
+      run: |
+        check_time.sh adjoint_sensitivities_non_optimal_parameters tests/performance/test.py adjoint_sensitivities_non_optimal_parameters

deps/AMICI/.gitrepo

@@ -5,8 +5,8 @@
 ;
 [subrepo]
 	remote = git@github.com:ICB-DCM/AMICI.git
-	branch = v0.10.13
-	commit = 76162dfc09d74d439939ab92eac7a2f7b80f0873
-	parent = 3fe186d2f265eb58466b7b0e25c56bc9c3a71abf
+	branch = v0.10.16
+	commit = a4e077f317d2d3eb6f93fcb1a849ddb4ffa7a175
+	parent = 6f77a9f618dc859380d4dcb6ee73676d2b388b62
 	cmdver = 0.4.0
 	method = merge

deps/AMICI/.travis.yml

@@ -2,6 +2,8 @@ branches:
   only:
     - master
     - develop
+    # Release branches, otherwise they cannot be merged into protected master
+    - /^release.*/
     # Version tags
     - /^v\d+\.\d+(\.\d+)?(-\S*)?$/
 
@@ -21,7 +23,6 @@ matrix:
             - libatlas-base-dev
             - lcov
             - libboost-serialization-dev
-            - swig3.0
             - g++-5
             - libc6-dbg
       env:
@@ -95,7 +96,8 @@ matrix:
 
 install:
     - export BASE_DIR=`pwd`
-    - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then mkdir -p ~/bin/ && ln -s /usr/bin/swig3.0 ~/bin/swig && export PATH=~/bin/:$PATH; fi # Python distutils only looks for `swig` and does not find `swig3.0`
+    # Build swig4.0 (not yet available with apt) to include pydoc in source distribution for pypi
+    - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then scripts/downloadAndBuildSwig.sh && export PATH=${BASE_DIR}/ThirdParty/swig-4.0.1/install/bin:${PATH}; fi
     - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then export PYTHON_EXECUTABLE=$(which python3); fi # cmake wont be able to find python3 on its own ...
     - if [[ "$TRAVIS_OS_NAME" == "linux" ]] && [[ "$CI_MODE" == "test" ]]; then pip3 install --upgrade pip==9.0.3 setuptools wheel pkgconfig scipy; fi
     - if [[ "$TRAVIS_OS_NAME" != "linux" ]] && [[ "$CI_MODE" == "test" ]]; then pip3 install --user --upgrade pip==9.0.3 setuptools wheel pkgconfig scipy; fi

deps/AMICI/INSTALL.md

@@ -52,6 +52,20 @@ and custom installations.
 For Python-AMICI usage see 
 [https://github.com/ICB-DCM/AMICI/blob/master/documentation/PYTHON.md](https://github.com/ICB-DCM/AMICI/blob/master/documentation/PYTHON.md).
 
+
+### Installation of development versions
+
+To install development versions which have not been released to pypi yet,
+you can install AMICI with pip directly from GitHub using:
+
+    pip3 install -e https://github.com/icb-dcm/amici/archive/develop.zip#egg=amici\&subdirectory=python/sdist
+
+Replace `develop` by the branch or commit you want to install.
+
+Note that this will probably not work on Windows which does not support
+symlinks by default
+(https://stackoverflow.com/questions/5917249/git-symlinks-in-windows/49913019#49913019).
+
 ### Light installation 
 
 In case you only want to use the AMICI Python package for generating model code
@@ -308,6 +322,10 @@ implementations such as Accelerate, Intel MKL, cblas, openblas, atlas.
 The matlab interface uses the MATLAB MKL, which requires no separate
 installation.
 
+On Ubuntu, this requirement can be satisfied with
+
+    apt install libatlas-base-dev
+
 #### C++ compiler
 
 All AMICI installations require a C++11-compatible C++ compiler.

deps/AMICI/README.md

@@ -128,6 +128,8 @@ However, the following features are unlikely to be supported:
 - initial assignments for parameters
 - `delay()` due to missing SUNDIALS solver support
 
+In addition to SBML, we also plan to implement support for the [Simulation Experiment Description Markup Language (SED-ML)](https://sed-ml.org/).
+
 
 ## Current build status
 

deps/AMICI/documentation/FAQ.md

@@ -42,3 +42,17 @@ __Q__: The simulation/sensitivities I get are incorrect.
 
 __A__: There are some known issues, especially with adjoint sensitivities, events and DAEs. If your particular problem is not featured in the [issues](https://github.com/ICB-DCM/AMICI/issues) list, please add it!
 
+---
+
+__Q__: I am trying to install the AMICI Python package, but installation fails
+with something like
+
+    amici/src/cblas.cpp:16:13: fatal error: cblas.h: No such file or directory
+    #include <cblas.h>
+             ^~~~~~~~~
+    compilation terminated.
+    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
+
+__A__: You will have to install a CBLAS-compatible BLAS library and/or set
+`BLAS_CFLAGS` as described in the [installation guide](INSTALL.md).
+