tutorials/train_test.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob, os, subprocess\n",
    "from IPython.display import IFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Precursors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train a model, you first need to convert your sequences and targets into the input HDF5 format. Check out my tutorials for how to do that; they're linked from the [main page](../README.md).\n",
    "\n",
    "For this tutorial, grab a small example HDF5 that I constructed here with 10% of the training sequences and only GM12878 targets for various DNase-seq, ChIP-seq, and CAGE experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "if len(glob.glob('data/heart_l131k/tfrecords/*.tfr')) == 0:\n",
    "    subprocess.call('curl -o data/heart_l131k.tgz https://storage.googleapis.com/basenji_tutorial_data/heart_l131k.tgz', shell=True)\n",
    "    subprocess.call('tar -xzvf data/heart_l131k.tgz', shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, you need to decide what sort of architecture to use. This grammar probably needs work; my goal was to enable hyperparameter searches to write the parameters to file so that I could run parallel training jobs to explore the hyperparameter space. I included an example set of parameters that will work well with this data in models/params_small.txt."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, run [basenji_train.py](https://github.com/calico/basenji/blob/master/bin/basenji_train.py) to train a model. The program will offer training feedback via stdout and write the model output files to the prefix given by the *-s* parameter.\n",
    "\n",
    "The most relevant options here are:\n",
    "\n",
    "| Option/Argument | Value | Note |\n",
    "|:---|:---|:---|\n",
    "| --augment_rc | True | Process even-numbered epochs as forward, odd-numbered as reverse complemented. |\n",
    "| --ensemble_rc | True | Average forward and reverse complemented predictions on validation set. |\n",
    "| --augment_shifts | \"1,0,-1\" | Rotate epochs over small sequence shifts. |\n",
    "| --logdir | models/heart | Directory to save training logs and model checkpoints. |\n",
    "| --params | models/params_small.txt | Table of parameters to setup the model architecture and optimization. |\n",
    "| --train_data | data/heart_l131k/tfrecords/train*.tfr | Training set TFRecords. |\n",
    "| --test_data | data/heart_l131k/tfrecords/valid*.tfr | Validation set TFRecords. |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to train, uncomment the following line and run it. Depending on your hardware, it may require several hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "//anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "2019-11-10 17:59:08.768155: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n",
      "2019-11-10 17:59:08.793392: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: __inference_Dataset_flat_map_read_one_file_37\n",
      "data/heart_l131k/tfrecords/train-*.tfr has 1499 sequences with 3/3 targets\n",
      "2019-11-10 17:59:10.581209: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: __inference_Dataset_flat_map_read_one_file_6087\n",
      "2019-11-10 17:59:10.632040: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: __inference_Dataset_flat_map_read_one_file_6157\n",
      "data/heart_l131k/tfrecords/valid-*.tfr has 180 sequences with 3/3 targets\n",
      "2019-11-10 17:59:10.874229: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: __inference_Dataset_flat_map_read_one_file_6929\n",
      "2019-11-10 17:59:10.986618: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_6978\n",
      "2019-11-10 17:59:10.986647: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_6979\n",
      "2019-11-10 17:59:11.250601: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_7038\n",
      "2019-11-10 17:59:11.250659: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_7039\n",
      "2019-11-10 17:59:11.255172: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_7019\n",
      "2019-11-10 17:59:11.255236: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_7038\n",
      "2019-11-10 17:59:11.255273: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_7039\n",
      "2019-11-10 17:59:11.255305: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_7020\n",
      "Model: \"model_1\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "sequence (InputLayer)           [(None, 131072, 4)]  0                                            \n",
      "__________________________________________________________________________________________________\n",
      "stochastic_reverse_complement ( ((None, 131072, 4),  0           sequence[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "stochastic_shift (StochasticShi (None, 131072, 4)    0           stochastic_reverse_complement[0][\n",
      "__________________________________________________________________________________________________\n",
      "gelu (GELU)                     (None, 131072, 4)    0           stochastic_shift[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "conv1d (Conv1D)                 (None, 131072, 64)   3840        gelu[0][0]                       \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization (BatchNorma (None, 131072, 64)   256         conv1d[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling1d (MaxPooling1D)    (None, 16384, 64)    0           batch_normalization[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "gelu_1 (GELU)                   (None, 16384, 64)    0           max_pooling1d[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_1 (Conv1D)               (None, 16384, 64)    20480       gelu_1[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, 16384, 64)    256         conv1d_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling1d_1 (MaxPooling1D)  (None, 4096, 64)     0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "gelu_2 (GELU)                   (None, 4096, 64)     0           max_pooling1d_1[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_2 (Conv1D)               (None, 4096, 72)     23040       gelu_2[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, 4096, 72)     288         conv1d_2[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling1d_2 (MaxPooling1D)  (None, 1024, 72)     0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "gelu_3 (GELU)                   (None, 1024, 72)     0           max_pooling1d_2[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_3 (Conv1D)               (None, 1024, 32)     6912        gelu_3[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, 1024, 32)     128         conv1d_3[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "gelu_4 (GELU)                   (None, 1024, 32)     0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_4 (Conv1D)               (None, 1024, 72)     2304        gelu_4[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, 1024, 72)     288         conv1d_4[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "dropout (Dropout)               (None, 1024, 72)     0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "add (Add)                       (None, 1024, 72)     0           max_pooling1d_2[0][0]            \n",
      "                                                                 dropout[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "gelu_5 (GELU)                   (None, 1024, 72)     0           add[0][0]                        \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_5 (Conv1D)               (None, None, 32)     6912        gelu_5[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, None, 32)     128         conv1d_5[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "gelu_6 (GELU)                   (None, None, 32)     0           batch_normalization_5[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_6 (Conv1D)               (None, None, 72)     2304        gelu_6[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_6 (BatchNor (None, None, 72)     288         conv1d_6[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, None, 72)     0           batch_normalization_6[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "add_1 (Add)                     (None, 1024, 72)     0           add[0][0]                        \n",
      "                                                                 dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "gelu_7 (GELU)                   (None, 1024, 72)     0           add_1[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_7 (Conv1D)               (None, None, 32)     6912        gelu_7[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_7 (BatchNor (None, None, 32)     128         conv1d_7[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "gelu_8 (GELU)                   (None, None, 32)     0           batch_normalization_7[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_8 (Conv1D)               (None, None, 72)     2304        gelu_8[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_8 (BatchNor (None, None, 72)     288         conv1d_8[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, None, 72)     0           batch_normalization_8[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "add_2 (Add)                     (None, 1024, 72)     0           add_1[0][0]                      \n",
      "                                                                 dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "gelu_9 (GELU)                   (None, 1024, 72)     0           add_2[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_9 (Conv1D)               (None, None, 32)     6912        gelu_9[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_9 (BatchNor (None, None, 32)     128         conv1d_9[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "gelu_10 (GELU)                  (None, None, 32)     0           batch_normalization_9[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_10 (Conv1D)              (None, None, 72)     2304        gelu_10[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_10 (BatchNo (None, None, 72)     288         conv1d_10[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, None, 72)     0           batch_normalization_10[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_3 (Add)                     (None, 1024, 72)     0           add_2[0][0]                      \n",
      "                                                                 dropout_3[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "gelu_11 (GELU)                  (None, 1024, 72)     0           add_3[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_11 (Conv1D)              (None, None, 32)     6912        gelu_11[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_11 (BatchNo (None, None, 32)     128         conv1d_11[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "gelu_12 (GELU)                  (None, None, 32)     0           batch_normalization_11[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_12 (Conv1D)              (None, None, 72)     2304        gelu_12[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_12 (BatchNo (None, None, 72)     288         conv1d_12[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_4 (Dropout)             (None, None, 72)     0           batch_normalization_12[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_4 (Add)                     (None, 1024, 72)     0           add_3[0][0]                      \n",
      "                                                                 dropout_4[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "gelu_13 (GELU)                  (None, 1024, 72)     0           add_4[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_13 (Conv1D)              (None, None, 32)     6912        gelu_13[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_13 (BatchNo (None, None, 32)     128         conv1d_13[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "gelu_14 (GELU)                  (None, None, 32)     0           batch_normalization_13[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_14 (Conv1D)              (None, None, 72)     2304        gelu_14[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_14 (BatchNo (None, None, 72)     288         conv1d_14[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_5 (Dropout)             (None, None, 72)     0           batch_normalization_14[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_5 (Add)                     (None, 1024, 72)     0           add_4[0][0]                      \n",
      "                                                                 dropout_5[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "gelu_15 (GELU)                  (None, 1024, 72)     0           add_5[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_15 (Conv1D)              (None, 1024, 64)     4608        gelu_15[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_15 (BatchNo (None, 1024, 64)     256         conv1d_15[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_6 (Dropout)             (None, 1024, 64)     0           batch_normalization_15[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "gelu_16 (GELU)                  (None, 1024, 64)     0           dropout_6[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense (Dense)                   (None, 1024, 3)      195         gelu_16[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "switch_reverse (SwitchReverse)  (None, 1024, 3)      0           dense[0][0]                      \n",
      "                                                                 stochastic_reverse_complement[0][\n",
      "==================================================================================================\n",
      "Total params: 111,011\n",
      "Trainable params: 109,235\n",
      "Non-trainable params: 1,776\n",
      "__________________________________________________________________________________________________\n",
      "None\n",
      "Epoch 1/20\n",
      "2019-11-10 17:59:12.530932: W ./tensorflow/core/framework/model.h:213] Encountered a stop event that was not preceded by a start event.\n",
      "WARNING:tensorflow:From //anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/clip_ops.py:157: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "2019-11-10 17:59:22.030164: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.4433 - pearsonr: 0.1656 - r2: 0.0206\n",
      "Epoch 00001: val_loss improved from inf to 0.38308, saving model to models/heart/model_best.h5\n",
      "374/374 [==============================] - 257s 688ms/step - loss: 0.4433 - pearsonr: 0.1659 - r2: 0.0207 - val_loss: 0.3831 - val_pearsonr: 0.4126 - val_r2: 0.1556\n",
      "Epoch 2/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3876 - pearsonr: 0.2163 - r2: 0.0432\n",
      "Epoch 00002: val_loss improved from 0.38308 to 0.37163, saving model to models/heart/model_best.h5\n",
      "374/374 [==============================] - 251s 672ms/step - loss: 0.3875 - pearsonr: 0.2164 - r2: 0.0433 - val_loss: 0.3716 - val_pearsonr: 0.4074 - val_r2: 0.1034\n",
      "Epoch 3/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3811 - pearsonr: 0.2263 - r2: 0.0417\n",
      "Epoch 00003: val_loss did not improve from 0.37163\n",
      "374/374 [==============================] - 259s 692ms/step - loss: 0.3810 - pearsonr: 0.2263 - r2: 0.0416 - val_loss: 0.4004 - val_pearsonr: 0.4259 - val_r2: 0.0473\n",
      "Epoch 4/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3723 - pearsonr: 0.2450 - r2: 0.0563\n",
      "Epoch 00004: val_loss did not improve from 0.37163\n",
      "374/374 [==============================] - 255s 683ms/step - loss: 0.3721 - pearsonr: 0.2455 - r2: 0.0565 - val_loss: 0.3778 - val_pearsonr: 0.4443 - val_r2: 0.1897\n",
      "Epoch 5/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3640 - pearsonr: 0.2704 - r2: 0.0726\n",
      "Epoch 00005: val_loss did not improve from 0.37163\n",
      "374/374 [==============================] - 259s 693ms/step - loss: 0.3641 - pearsonr: 0.2704 - r2: 0.0726 - val_loss: 0.3732 - val_pearsonr: 0.4495 - val_r2: 0.1897\n",
      "Epoch 6/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3560 - pearsonr: 0.2924 - r2: 0.0853\n",
      "Epoch 00006: val_loss did not improve from 0.37163\n",
      "374/374 [==============================] - 257s 688ms/step - loss: 0.3562 - pearsonr: 0.2926 - r2: 0.0854 - val_loss: 0.3751 - val_pearsonr: 0.4572 - val_r2: 0.1946\n",
      "Epoch 7/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3573 - pearsonr: 0.3025 - r2: 0.0914\n",
      "Epoch 00007: val_loss did not improve from 0.37163\n",
      "374/374 [==============================] - 261s 698ms/step - loss: 0.3574 - pearsonr: 0.3027 - r2: 0.0915 - val_loss: 0.3723 - val_pearsonr: 0.4576 - val_r2: 0.2036\n",
      "Epoch 8/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3582 - pearsonr: 0.2886 - r2: 0.0810\n",
      "Epoch 00008: val_loss did not improve from 0.37163\n",
      "374/374 [==============================] - 262s 700ms/step - loss: 0.3583 - pearsonr: 0.2885 - r2: 0.0810 - val_loss: 0.3858 - val_pearsonr: 0.4383 - val_r2: 0.1887\n",
      "Epoch 9/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3491 - pearsonr: 0.3051 - r2: 0.0932\n",
      "Epoch 00009: val_loss improved from 0.37163 to 0.36718, saving model to models/heart/model_best.h5\n",
      "374/374 [==============================] - 264s 707ms/step - loss: 0.3491 - pearsonr: 0.3051 - r2: 0.0932 - val_loss: 0.3672 - val_pearsonr: 0.4285 - val_r2: 0.1713\n",
      "Epoch 10/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3517 - pearsonr: 0.3122 - r2: 0.0955\n",
      "Epoch 00010: val_loss improved from 0.36718 to 0.35215, saving model to models/heart/model_best.h5\n",
      "374/374 [==============================] - 258s 690ms/step - loss: 0.3517 - pearsonr: 0.3123 - r2: 0.0956 - val_loss: 0.3522 - val_pearsonr: 0.4528 - val_r2: 0.1807\n",
      "Epoch 11/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3502 - pearsonr: 0.3098 - r2: 0.0959\n",
      "Epoch 00011: val_loss did not improve from 0.35215\n",
      "374/374 [==============================] - 257s 688ms/step - loss: 0.3502 - pearsonr: 0.3099 - r2: 0.0960 - val_loss: 0.3875 - val_pearsonr: 0.3933 - val_r2: 0.0193\n",
      "Epoch 12/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3455 - pearsonr: 0.3314 - r2: 0.1098\n",
      "Epoch 00012: val_loss did not improve from 0.35215\n",
      "374/374 [==============================] - 256s 686ms/step - loss: 0.3458 - pearsonr: 0.3312 - r2: 0.1097 - val_loss: 0.3792 - val_pearsonr: 0.4253 - val_r2: 0.1468\n",
      "Epoch 13/20\n",
      "373/374 [============================>.] - ETA: 1s - loss: 0.3520 - pearsonr: 0.3204 - r2: 0.1015\n",
      "Epoch 00013: val_loss did not improve from 0.35215\n",
      "374/374 [==============================] - 497s 1s/step - loss: 0.3522 - pearsonr: 0.3202 - r2: 0.1015 - val_loss: 0.3632 - val_pearsonr: 0.4507 - val_r2: 0.0719\n",
      "Epoch 14/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3426 - pearsonr: 0.3551 - r2: 0.1257\n",
      "Epoch 00014: val_loss improved from 0.35215 to 0.34557, saving model to models/heart/model_best.h5\n",
      "374/374 [==============================] - 258s 691ms/step - loss: 0.3426 - pearsonr: 0.3554 - r2: 0.1259 - val_loss: 0.3456 - val_pearsonr: 0.4801 - val_r2: 0.2311\n",
      "Epoch 15/20\n",
      "373/374 [============================>.] - ETA: 18s - loss: 0.3378 - pearsonr: 0.3843 - r2: 0.1459\n",
      "Epoch 00015: val_loss did not improve from 0.34557\n",
      "374/374 [==============================] - 6723s 18s/step - loss: 0.3378 - pearsonr: 0.3843 - r2: 0.1459 - val_loss: 0.3495 - val_pearsonr: 0.4716 - val_r2: 0.1875\n",
      "Epoch 16/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3265 - pearsonr: 0.4003 - r2: 0.1594\n",
      "Epoch 00016: val_loss did not improve from 0.34557\n",
      "374/374 [==============================] - 260s 695ms/step - loss: 0.3266 - pearsonr: 0.4003 - r2: 0.1594 - val_loss: 0.3712 - val_pearsonr: 0.4846 - val_r2: 0.2337\n",
      "Epoch 17/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3253 - pearsonr: 0.3961 - r2: 0.1551\n",
      "Epoch 00017: val_loss did not improve from 0.34557\n",
      "374/374 [==============================] - 261s 697ms/step - loss: 0.3248 - pearsonr: 0.3970 - r2: 0.1559 - val_loss: 0.3550 - val_pearsonr: 0.4630 - val_r2: 0.0185\n",
      "Epoch 18/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3246 - pearsonr: 0.4538 - r2: 0.2030\n",
      "Epoch 00018: val_loss did not improve from 0.34557\n",
      "374/374 [==============================] - 260s 695ms/step - loss: 0.3248 - pearsonr: 0.4535 - r2: 0.2028 - val_loss: 0.3508 - val_pearsonr: 0.4651 - val_r2: 0.2215\n",
      "Epoch 19/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3218 - pearsonr: 0.4874 - r2: 0.2272\n",
      "Epoch 00019: val_loss did not improve from 0.34557\n",
      "374/374 [==============================] - 262s 700ms/step - loss: 0.3217 - pearsonr: 0.4872 - r2: 0.2272 - val_loss: 0.3603 - val_pearsonr: 0.4710 - val_r2: 0.1676\n",
      "Epoch 20/20\n",
      "373/374 [============================>.] - ETA: 0s - loss: 0.3310 - pearsonr: 0.4510 - r2: 0.2015\n",
      "Epoch 00020: val_loss did not improve from 0.34557\n",
      "374/374 [==============================] - 259s 693ms/step - loss: 0.3303 - pearsonr: 0.4494 - r2: 0.1999 - val_loss: 0.3972 - val_pearsonr: 0.4722 - val_r2: 0.2000\n"
     ]
    }
   ],
   "source": [
    "! basenji_train.py -o models/heart models/params_small.json data/heart_l131k"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you can just download a trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.isdir('models/heart'):\n",
    "    os.mkdir('models/heart')\n",
    "if not os.path.isfile('models/heart/model_best.tf.meta'):\n",
    "    subprocess.call('curl -o models/heart/model_best.tf.index https://storage.googleapis.com/basenji_tutorial_data/model_best.tf.index', shell=True)\n",
    "    subprocess.call('curl -o models/heart/model_best.tf.meta https://storage.googleapis.com/basenji_tutorial_data/model_best.tf.meta', shell=True)\n",
    "    subprocess.call('curl -o models/heart/model_best.tf.data-00000-of-00001 https://storage.googleapis.com/basenji_tutorial_data/model_best.tf.data-00000-of-00001', shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "models/heart/model_best.tf will now specify the name of your saved model to be provided to other programs.\n",
    "\n",
    "To further benchmark the accuracy (e.g. computing significant \"peak\" accuracy), use [basenji_test.py](https://github.com/calico/basenji/blob/master/bin/basenji_test.py).\n",
    "\n",
    "The most relevant options here are:\n",
    "\n",
    "| Option/Argument | Value | Note |\n",
    "|:---|:---|:---|\n",
    "| --rc | | Average the forward and reverse complement to form prediction. |\n",
    "| -o | output/heart_test | Output directory. |\n",
    "| --ai | 0,1,2 | Make accuracy scatter plots for targets 0, 1, and 2. |\n",
    "| --ti | 3,4,5 | Make BigWig tracks for targets 3, 4, and 5. |\n",
    "| -t | data/heart_l131k.bed | BED file describing sequence regions for BigWig track output. |\n",
    "| params_file | models/params_small.txt | Table of parameters to setup the model architecture and optimization. |\n",
    "| model_file | models/heart/model_best.tf | Trained saved model prefix. |\n",
    "| data_file | data/heart_l131k.h5 | HDF5 file containing the test input and output datasets as generated by [basenji_hdf5_single.py](https://github.com/calico/basenji/blob/master/bin/basenji_hdf5_single.py) |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "//anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "//anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "2019-11-10 21:45:43.530838: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n",
      "2019-11-10 21:45:43.572807: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: __inference_Dataset_flat_map_read_one_file_37\n",
      "data/heart_l131k/tfrecords/test-*.tfr has 179 sequences with 3/3 targets\n",
      "2019-11-10 21:45:43.855273: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: __inference_Dataset_flat_map_read_one_file_805\n",
      "2019-11-10 21:45:43.971565: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_854\n",
      "2019-11-10 21:45:43.971595: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_855\n",
      "2019-11-10 21:45:44.222221: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_914\n",
      "2019-11-10 21:45:44.222282: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_915\n",
      "2019-11-10 21:45:44.226511: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_895\n",
      "2019-11-10 21:45:44.226592: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_914\n",
      "2019-11-10 21:45:44.226642: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_915\n",
      "2019-11-10 21:45:44.226673: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_896\n",
      "Model: \"model_1\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "sequence (InputLayer)           [(None, 131072, 4)]  0                                            \n",
      "__________________________________________________________________________________________________\n",
      "stochastic_reverse_complement ( ((None, 131072, 4),  0           sequence[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "stochastic_shift (StochasticShi (None, 131072, 4)    0           stochastic_reverse_complement[0][\n",
      "__________________________________________________________________________________________________\n",
      "gelu (GELU)                     (None, 131072, 4)    0           stochastic_shift[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "conv1d (Conv1D)                 (None, 131072, 64)   3840        gelu[0][0]                       \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization (BatchNorma (None, 131072, 64)   256         conv1d[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling1d (MaxPooling1D)    (None, 16384, 64)    0           batch_normalization[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "gelu_1 (GELU)                   (None, 16384, 64)    0           max_pooling1d[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_1 (Conv1D)               (None, 16384, 64)    20480       gelu_1[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, 16384, 64)    256         conv1d_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling1d_1 (MaxPooling1D)  (None, 4096, 64)     0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "gelu_2 (GELU)                   (None, 4096, 64)     0           max_pooling1d_1[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_2 (Conv1D)               (None, 4096, 72)     23040       gelu_2[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, 4096, 72)     288         conv1d_2[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "max_pooling1d_2 (MaxPooling1D)  (None, 1024, 72)     0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "gelu_3 (GELU)                   (None, 1024, 72)     0           max_pooling1d_2[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_3 (Conv1D)               (None, 1024, 32)     6912        gelu_3[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, 1024, 32)     128         conv1d_3[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "gelu_4 (GELU)                   (None, 1024, 32)     0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_4 (Conv1D)               (None, 1024, 72)     2304        gelu_4[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, 1024, 72)     288         conv1d_4[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "dropout (Dropout)               (None, 1024, 72)     0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "add (Add)                       (None, 1024, 72)     0           max_pooling1d_2[0][0]            \n",
      "                                                                 dropout[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "gelu_5 (GELU)                   (None, 1024, 72)     0           add[0][0]                        \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_5 (Conv1D)               (None, None, 32)     6912        gelu_5[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, None, 32)     128         conv1d_5[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "gelu_6 (GELU)                   (None, None, 32)     0           batch_normalization_5[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_6 (Conv1D)               (None, None, 72)     2304        gelu_6[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_6 (BatchNor (None, None, 72)     288         conv1d_6[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, None, 72)     0           batch_normalization_6[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "add_1 (Add)                     (None, 1024, 72)     0           add[0][0]                        \n",
      "                                                                 dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "gelu_7 (GELU)                   (None, 1024, 72)     0           add_1[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_7 (Conv1D)               (None, None, 32)     6912        gelu_7[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_7 (BatchNor (None, None, 32)     128         conv1d_7[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "gelu_8 (GELU)                   (None, None, 32)     0           batch_normalization_7[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_8 (Conv1D)               (None, None, 72)     2304        gelu_8[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_8 (BatchNor (None, None, 72)     288         conv1d_8[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, None, 72)     0           batch_normalization_8[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "add_2 (Add)                     (None, 1024, 72)     0           add_1[0][0]                      \n",
      "                                                                 dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "gelu_9 (GELU)                   (None, 1024, 72)     0           add_2[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_9 (Conv1D)               (None, None, 32)     6912        gelu_9[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_9 (BatchNor (None, None, 32)     128         conv1d_9[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "gelu_10 (GELU)                  (None, None, 32)     0           batch_normalization_9[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_10 (Conv1D)              (None, None, 72)     2304        gelu_10[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_10 (BatchNo (None, None, 72)     288         conv1d_10[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, None, 72)     0           batch_normalization_10[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_3 (Add)                     (None, 1024, 72)     0           add_2[0][0]                      \n",
      "                                                                 dropout_3[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "gelu_11 (GELU)                  (None, 1024, 72)     0           add_3[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_11 (Conv1D)              (None, None, 32)     6912        gelu_11[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_11 (BatchNo (None, None, 32)     128         conv1d_11[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "gelu_12 (GELU)                  (None, None, 32)     0           batch_normalization_11[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_12 (Conv1D)              (None, None, 72)     2304        gelu_12[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_12 (BatchNo (None, None, 72)     288         conv1d_12[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_4 (Dropout)             (None, None, 72)     0           batch_normalization_12[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_4 (Add)                     (None, 1024, 72)     0           add_3[0][0]                      \n",
      "                                                                 dropout_4[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "gelu_13 (GELU)                  (None, 1024, 72)     0           add_4[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_13 (Conv1D)              (None, None, 32)     6912        gelu_13[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_13 (BatchNo (None, None, 32)     128         conv1d_13[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "gelu_14 (GELU)                  (None, None, 32)     0           batch_normalization_13[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_14 (Conv1D)              (None, None, 72)     2304        gelu_14[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_14 (BatchNo (None, None, 72)     288         conv1d_14[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_5 (Dropout)             (None, None, 72)     0           batch_normalization_14[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "add_5 (Add)                     (None, 1024, 72)     0           add_4[0][0]                      \n",
      "                                                                 dropout_5[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "gelu_15 (GELU)                  (None, 1024, 72)     0           add_5[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_15 (Conv1D)              (None, 1024, 64)     4608        gelu_15[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_15 (BatchNo (None, 1024, 64)     256         conv1d_15[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_6 (Dropout)             (None, 1024, 64)     0           batch_normalization_15[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "gelu_16 (GELU)                  (None, 1024, 64)     0           dropout_6[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense (Dense)                   (None, 1024, 3)      195         gelu_16[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "switch_reverse (SwitchReverse)  (None, 1024, 3)      0           dense[0][0]                      \n",
      "                                                                 stochastic_reverse_complement[0][\n",
      "==================================================================================================\n",
      "Total params: 111,011\n",
      "Trainable params: 109,235\n",
      "Non-trainable params: 1,776\n",
      "__________________________________________________________________________________________________\n",
      "None\n",
      "2019-11-10 21:45:45.437792: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_3594\n",
      "2019-11-10 21:45:45.437821: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_3595\n",
      "2019-11-10 21:45:45.539367: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_3650\n",
      "2019-11-10 21:45:45.539471: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_3651\n",
      "2019-11-10 21:45:45.547124: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_3631\n",
      "2019-11-10 21:45:45.547229: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_3650\n",
      "2019-11-10 21:45:45.547291: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_3651\n",
      "2019-11-10 21:45:45.547345: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_3632\n",
      "2019-11-10 21:45:46.200085: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_5178\n",
      "2019-11-10 21:45:46.200114: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_5179\n",
      "2019-11-10 21:45:46.229871: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_5234\n",
      "2019-11-10 21:45:46.229926: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_5235\n",
      "2019-11-10 21:45:46.234356: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_5215\n",
      "2019-11-10 21:45:46.234419: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_5234\n",
      "2019-11-10 21:45:46.234457: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_5235\n",
      "2019-11-10 21:45:46.234491: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_5216\n",
      "2019-11-10 21:45:46.966716: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_6762\n",
      "2019-11-10 21:45:46.966746: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_6763\n",
      "2019-11-10 21:45:46.997261: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_6818\n",
      "2019-11-10 21:45:46.997321: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_6819\n",
      "2019-11-10 21:45:47.001883: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_6799\n",
      "2019-11-10 21:45:47.001951: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_6818\n",
      "2019-11-10 21:45:47.001988: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_6819\n",
      "2019-11-10 21:45:47.002022: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_6800\n",
      "2019-11-10 21:45:47.642678: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_8346\n",
      "2019-11-10 21:45:47.642709: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_8347\n",
      "2019-11-10 21:45:47.673226: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_8402\n",
      "2019-11-10 21:45:47.673285: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_8403\n",
      "2019-11-10 21:45:47.677688: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_8383\n",
      "2019-11-10 21:45:47.677819: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_8402\n",
      "2019-11-10 21:45:47.677873: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_8403\n",
      "2019-11-10 21:45:47.677907: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_8384\n",
      "2019-11-10 21:45:48.439626: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_9930\n",
      "2019-11-10 21:45:48.439657: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_9931\n",
      "2019-11-10 21:45:48.468684: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_9986\n",
      "2019-11-10 21:45:48.468737: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_9987\n",
      "2019-11-10 21:45:48.472988: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_9967\n",
      "2019-11-10 21:45:48.473053: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_9986\n",
      "2019-11-10 21:45:48.473091: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_9987\n",
      "2019-11-10 21:45:48.473125: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_9968\n",
      "2019-11-10 21:45:49.111117: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_11514\n",
      "2019-11-10 21:45:49.111146: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_11515\n",
      "2019-11-10 21:45:49.140789: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_11570\n",
      "2019-11-10 21:45:49.140846: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_11571\n",
      "2019-11-10 21:45:49.145178: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_11551\n",
      "2019-11-10 21:45:49.145242: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_true_11570\n",
      "2019-11-10 21:45:49.145278: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_11571\n",
      "2019-11-10 21:45:49.145310: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: cond_false_11552\n",
      "     45/Unknown - 42s 927ms/step - loss: 0.3288 - pearsonr: 0.4365 - r2: 0.1875\n",
      "Test Loss:         0.32880\n",
      "Test R2:           0.18752\n",
      "Test PearsonR:     0.43646\n",
      "2019-11-10 21:47:13.574264: W tensorflow/core/common_runtime/eager/context.cc:371] Added two functions with the same name: __inference_Dataset_flat_map_read_one_file_25395\n"
     ]
    }
   ],
   "source": [
    "! basenji_test.py --ai 0,1,2 -o output/heart_test --rc --shifts \"1,0,-1\" models/params_small.json models/heart/model_best.h5 data/heart_l131k"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*data/heart_test/acc.txt* is a table specifiying the loss function value, R2, R2 after log2, and Spearman correlation for each dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "index\tr2\tpearsonr\tidentifier\tdescription\n",
      "0\t0.13260\t0.36457\tCNhs11760\taorta\n",
      "1\t0.23704\t0.50558\tCNhs12843\tartery\n",
      "2\t0.19291\t0.43924\tCNhs12856\tpulmonic_valve\n"
     ]
    }
   ],
   "source": [
    "! cat output/heart_test/acc.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The directories *pr*, *roc*, *violin*, and *scatter* in *data/heart_test* contain plots for the targets indexed by 0, 1, and 2 as specified by the --ai option above.\n",
    "\n",
    "E.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"600\"\n",
       "            height=\"500\"\n",
       "            src=\"output/heart_test/pr/t0.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7fc3108fc978>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame('output/heart_test/pr/t0.pdf', width=600, height=500)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}