diff --git a/Chiron+weight_visual/README.md b/Chiron+weight_visual/README.md deleted file mode 100644 index 193205f..0000000 --- a/Chiron+weight_visual/README.md +++ /dev/null @@ -1,174 +0,0 @@ -# Chiron -## A basecaller for Oxford Nanopore Technologies' sequencers -Using a deep learning CNN+RNN+CTC structure to establish end-to-end basecalling for the nanopore sequencer. -Built with **TensorFlow** and python 2.7. - -If you found Chiron useful, please consider to cite: -> Teng, H., et al. (2017). Chiron: Translating nanopore raw signal directly into nucleotide sequence using deep learning. [bioRxiv 179531] (https://www.biorxiv.org/content/early/2017/09/12/179531) - ---- - -## Install -### Install using `pip` (recommended) -If you currently have TensorFlow installed on your system, we would advise you to create a virtual environment to install Chiron into, this way there is no clash of versions etc. - -If you would like to do this, the best options would be [`virtualenv`](https://virtualenv.pypa.io/en/stable/installation/), the more user-friendly [`virtualenvwrapper`](https://virtualenvwrapper.readthedocs.io/en/latest/install.html), or through [anaconda](https://docs.continuum.io/anaconda/install/). After installing one of these and activating the virtual environment you will be installing Chiron into, continue with the rest of the installation instructions as normal. - -To install with `pip`: - -``` -pip install chiron -``` -This will install Chiron, the CPU-only distribution of TensorFlow (and it's dependencies), and [`h5py`](https://github.com/h5py/h5py) (required for reading in `.fast5` files). - -**Note**: If you are after the GPU version, follow the steps in the following section. - -### Install from GitHub -This is currently the best install method if you are wanting to run Chiron on in GPU mode (`pip install` version is coming). -``` -git clone https://github.com/haotianteng/chiron.git -cd chiron -``` -You will also need to install dependencies. - -For CPU-version: -``` -pip install tensorflow==1.0.1 -pip install h5py -``` -For GPU-version(Nvidia GPU required): -``` -pip install tensorflow-gpu==1.0.1 -pip install h5py -``` - -For alternate/detailed installation instructions for TensorFlow, see their [fantastic documentation](https://www.tensorflow.org/). - -## Basecall -### If installed from `pip`: -An example call to Chiron to run basecalling is: -``` -chiron call -i -o - -``` - -### If installed from GitHub: - -All Chiron functionality can be run from **entry.py** in the Chiron folder. (You might like to also add the path to Chiron into your path for ease of running). - -``` -python chiron/entry.py call -i -o - -``` - -### Test run - -We provide 5 sample fast5 files (courtesy of [nanonet](https://github.com/nanoporetech/nanonet)) in the GitHub repository which you can run a test on. These are located in `chiron/example_data/`. From inside the Chiron repository: -``` -python chiron/entry.py call -i chiron/example_folder/ -o -``` - - -### Output -`chiron call` will create five folders in `` called `raw`, `result`, `segments`, `meta`, and `reference`. - -* `result`: fastq/fasta files with the same name as the fast5 file they contain the basecalling result for. To create a single, merged version of these fasta files, try something like `paste --delimiter=\\n --serial result/*.fasta > merged.fasta` -* `raw`: Contains a file for each fast5 file with it's raw signal. This file format is an list of integers. i.e `544 554 556 571 563 472 467 487 482 513 517 521 495 504 500 520 492 506 ... ` -* `segments`: Contains the segments basecalled from each fast5 file. -* `meta`: Contains the meta information for each read (read length, basecalling rate etc.). Each file has the same name as it's fast5 file. -* `reference`: Contains the reference sequence (if any). - -### Output format -With -e flag to output fastq file(default) with quality score or fasta file. -Example: -chiron call -i -o -e fastq - - -chiron call -i -o -e fasta - -## Training -The default DNA model trained on R9.4 protocol with a mix of Lambda and E.coli dataset, if the basecalling result is not satisfying, you can train a model on your own training data set. - -#### Hardware request: -Recommend training on GPU with TensorFlow - usually 8GB RAM (GPU) is required. - -#### Prepare the training data set. -Using raw.py script to extract the signal and label from the re-squiggled fast5 file. -(For how to re-squiggle fast5 file, check [here, nanoraw re-squiggle](https://nanoraw.readthedocs.io/en/latest/resquiggle.html#example-commands)) - -#### If installed from `pip`: -``` -chiron export -i -o -``` - -or directly use the raw.py script in utils. - -``` -python chiron/utils/raw.py --input --output -``` -`.signal` file and correspond `.label` file, a typical file format: - -`.signal` file format: -`544 554 556 571 563 472 467 487 482 513 517 521 495 504 500 520 492 506 ...` -i.e the file must contain only one row/column of raw signal numbers. - -`.label` file format: -``` -70 174 A -174 184 T -184 192 A -192 195 G -195 204 C -204 209 A -209 224 C -... -``` - -Each line represents a DNA base pair in the Pore. -* 1st column: Start position of the current nucleotide, position related to the signal vector (index count starts from zero). -* 2nd column: End position of the current nucleotide. -* 3rd column: Nucleotide, for DNA: A, G, C, or T. Although, there is no reason you could not use other labels. - -#### Adjust Chiron parameters -Go in to `chiron/chiron_rcnn_train.py` and change the hyper parameters in the `FLAGS` class. - -```py -class Flags(): - def __init__(self): - self.home_dir = "/home/haotianteng/UQ/deepBNS/" - self.data_dir = self.home_dir + 'data/Lambda_R9.4/raw/' - self.log_dir = self.home_dir+'/chiron/log/' - self.sequence_len = 200 - self.batch_size = 100 - self.step_rate = 1e-3 - self.max_steps = 2500 - self.k_mer = 1 - self.model_name = 'crnn5+5_res_moving_norm' - self.retrain = False -``` - -`data_dir`: The folder containing your signal and label files. -`log_dir`: The folder where you want to save the model. -`sequence_len`: The length of the segment you want to separate the sequence into. Longer length requires larger RAM. -`batch_size`: The batch size. -`step_rate`: Learning rate of the optimizer. -`max_step`: Maximum step of the optimizer. -`k_mer`: Chiron supports learning based on k-mer instead of a single nucleotide, this should be an odd number, even numbers will cause an error. -`model_name`: The name of the model. The record will be stored in the directory `log_dir/model_name/` -`retrain`: If this is a new model, or you want to load the model you trained before. The model will be loaded from `log_dir/model_name/` - -### Train - -``` -source activate tensorflow -``` -#### If installed from `pip`: -``` -chiron train --data_dir --log_dir --model_name -``` - -or run directly by - -``` -python chiron/chiron_rcnn_train.py -``` diff --git a/README.md b/README.md index c9a778f..7a09f08 100644 --- a/README.md +++ b/README.md @@ -29,18 +29,29 @@ we modified the docker from `https://github.com/anurag/fastai-course-1.git` ``` -nvidia-docker run -it \ +DATADIR=/data/nanopore + +nvidia-docker run \ + --rm -it \ --entrypoint /bin/zsh \ - -v /data/nanopore/new/fast5Dir/:/data \ - -p 8889:8888 \ + -v $DATADIR:/data \ + -p 8890:8888 \ + --name haruhi \ + -w /home/docker \ etheleon/chiron ``` To train deepore we need to run chiron_rcnn_train.py ``` -cd $HOME -python Chiron/chiron/chiron_rcnn_train.py +export CUDA_VISIBLE_DEVICES="1" +python deepore_SRU+swish/chiron/chiron_rcnn_train.py +``` + +If u want to log weights and the gradients u can use the log flag. + +``` +python deepore_SRU+swish/chiron/chiron__rcnn_train.py --logs 1 ``` # Reference diff --git a/Chiron+weight_visual/LICENSE.md b/deepore_SRU+Swish/LICENSE.md similarity index 100% rename from Chiron+weight_visual/LICENSE.md rename to deepore_SRU+Swish/LICENSE.md diff --git a/Chiron+weight_visual/MANIFEST.in b/deepore_SRU+Swish/MANIFEST.in similarity index 100% rename from Chiron+weight_visual/MANIFEST.in rename to deepore_SRU+Swish/MANIFEST.in diff --git a/deepore_SRU+Swish/README.md b/deepore_SRU+Swish/README.md new file mode 100644 index 0000000..585ae84 --- /dev/null +++ b/deepore_SRU+Swish/README.md @@ -0,0 +1,2 @@ +# Deepore +This modified version of Chiron allows us to visualise both the weights and the gradients to address training problem in Chiron, like local minima and bottlenecks during training that we have observed. diff --git a/Chiron+weight_visual/chiron/__init__.py b/deepore_SRU+Swish/chiron/__init__.py similarity index 100% rename from Chiron+weight_visual/chiron/__init__.py rename to deepore_SRU+Swish/chiron/__init__.py diff --git a/Chiron+weight_visual/chiron/chiron_cnn_train.py b/deepore_SRU+Swish/chiron/chiron_cnn_train.py similarity index 100% rename from Chiron+weight_visual/chiron/chiron_cnn_train.py rename to deepore_SRU+Swish/chiron/chiron_cnn_train.py diff --git a/Chiron+weight_visual/chiron/chiron_eval.py b/deepore_SRU+Swish/chiron/chiron_eval.py similarity index 100% rename from Chiron+weight_visual/chiron/chiron_eval.py rename to deepore_SRU+Swish/chiron/chiron_eval.py diff --git a/Chiron+weight_visual/chiron/chiron_input.py b/deepore_SRU+Swish/chiron/chiron_input.py similarity index 100% rename from Chiron+weight_visual/chiron/chiron_input.py rename to deepore_SRU+Swish/chiron/chiron_input.py diff --git a/deepore_SRU+Swish/chiron/chiron_input.pyc b/deepore_SRU+Swish/chiron/chiron_input.pyc new file mode 100644 index 0000000..ba76d3f Binary files /dev/null and b/deepore_SRU+Swish/chiron/chiron_input.pyc differ diff --git a/Chiron+weight_visual/chiron/chiron_rcnn_train.py b/deepore_SRU+Swish/chiron/chiron_rcnn_train.py similarity index 82% rename from Chiron+weight_visual/chiron/chiron_rcnn_train.py rename to deepore_SRU+Swish/chiron/chiron_rcnn_train.py index fe743f1..3e68754 100644 --- a/Chiron+weight_visual/chiron/chiron_rcnn_train.py +++ b/deepore_SRU+Swish/chiron/chiron_rcnn_train.py @@ -5,6 +5,8 @@ @author: haotianteng """ +import time,os +import argparse import tensorflow as tf from distutils.dir_util import copy_tree from chiron_input import read_raw_data_sets @@ -12,16 +14,16 @@ #from cnn import getcnnlogit #from rnn import rnn_layers from rnn import rnn_layers_one_direction -import time,os +from summary import variable_summaries def save_model(): copy_tree(os.path.dirname(os.path.abspath(__file__)),FLAGS.log_dir+FLAGS.model_name+'/model') -def inference(x,seq_length,training): - cnn_feature = getcnnfeature(x,training = training) +def inference(x,seq_length,training,verbose=False): + cnn_feature = getcnnfeature(x,training = training,verbose=verbose) feashape = cnn_feature.get_shape().as_list() ratio = FLAGS.sequence_len/feashape[1] # logits = rnn_layers(cnn_feature,seq_length/ratio,training,class_n = 4**FLAGS.k_mer+1 ) - logits = rnn_layers_one_direction(cnn_feature,seq_length/ratio,training,class_n = 4**FLAGS.k_mer+1 ) + logits = rnn_layers_one_direction(cnn_feature,seq_length/ratio,training,class_n = 4**FLAGS.k_mer+1,verbose=verbose ) # logits = getcnnlogit(cnn_feature) return logits,ratio @@ -31,11 +33,15 @@ def loss(logits,seq_len,label): tf.summary.scalar('loss',loss) return loss -def train_step(loss,global_step = None): - opt = tf.train.AdamOptimizer(FLAGS.step_rate).minimize(loss,global_step=global_step) +def train_step(loss,global_step = None,verbose=False): + opt = tf.train.AdamOptimizer(FLAGS.step_rate) # opt = tf.train.GradientDescentOptimizer(FLAGS.step_rate).minimize(loss) # opt = tf.train.RMSPropOptimizer(FLAGS.step_rate).minimize(loss) # opt = tf.train.MomentumOptimizer(FLAGS.step_rate,0.9).minimize(loss) + if verbose : + grad = opt.compute_gradients(loss) + tf.summary.scalar('grad',tf.reduce_mean(grad[0][0])) + opt = opt.minimize(loss,global_step=global_step) return opt def prediction(logits,seq_length,label,top_paths=1): """ @@ -58,6 +64,16 @@ def prediction(logits,seq_length,label,top_paths=1): return error def train(): + + parser = argparse.ArgumentParser() + parser.add_argument("--logs", help="logs") + args = parser.parse_args() + if args.logs: + print("logs turned on") + verbose = True + else: + print("logs turned off") + verbose = False training = tf.placeholder(tf.bool) global_step=tf.get_variable('global_step',trainable=False,shape=(),dtype = tf.int32,initializer = tf.zeros_initializer()) x = tf.placeholder(tf.float32,shape = [FLAGS.batch_size,FLAGS.sequence_len]) @@ -66,9 +82,9 @@ def train(): y_values = tf.placeholder(tf.int32) y_shape = tf.placeholder(tf.int64) y = tf.SparseTensor(y_indexs,y_values,y_shape) - logits,ratio = inference(x,seq_length,training) + logits,ratio = inference(x,seq_length,training,verbose) ctc_loss = loss(logits,seq_length,y) - opt = train_step(ctc_loss,global_step = global_step) + opt = train_step(ctc_loss,global_step = global_step,verbose=verbose) error = prediction(logits,seq_length,y) init = tf.global_variables_initializer() saver = tf.train.Saver() @@ -119,13 +135,14 @@ def run(args): if __name__ == "__main__": class Flags(): def __init__(self): - self.data_dir = '/media/haotianteng/Linux_ex/Nanopore_data/Lambda_R9.4/raw' - self.cache_dir = '/media/haotianteng/Linux_ex/Nanopore_data/Lambda_R9.4/cache' - self.log_dir = '/media/haotianteng/Linux_ex/GVM_model' + self.data_dir = '/home/docker/raw' #human + #self.data_dir = '/home/docker/ecoli' #ecoli + self.cache_dir = '/home/docker/out/cache' + self.log_dir = '/home/docker/out/logs' self.sequence_len = 300 - self.batch_size = 750 + self.batch_size = 64 self.step_rate = 1e-3 - self.max_steps = 20000 + self.max_steps = 10000 self.k_mer = 1 self.model_name = 'test' self.retrain =False diff --git a/Chiron+weight_visual/chiron/cnn.py b/deepore_SRU+Swish/chiron/cnn.py similarity index 92% rename from Chiron+weight_visual/chiron/cnn.py rename to deepore_SRU+Swish/chiron/cnn.py index a1a436b..833179f 100644 --- a/Chiron+weight_visual/chiron/cnn.py +++ b/deepore_SRU+Swish/chiron/cnn.py @@ -10,14 +10,15 @@ from tensorflow.contrib.layers import batch_norm from summary import variable_summaries -def conv_layer(indata,ksize,padding,training,name,dilate = 1,strides=[1,1,1,1],bias_term = False,active = True,BN= True): +def conv_layer(indata,ksize,padding,training,name,dilate = 1,strides=[1,1,1,1],bias_term = False,active = True,BN= True,verbose=False): """A standard convlotional layer""" with tf.variable_scope(name): W = tf.get_variable("weights", dtype = tf.float32, shape=ksize,initializer=tf.contrib.layers.xavier_initializer()) - variable_summaries(W) + beta = tf.get_variable("beta",dtype=tf.float32,shape=[1],initializer = tf.contrib.layers.xavier_initializer()) + if verbose : variable_summaries(W) if bias_term: b = tf.get_variable("bias", dtype=tf.float32,shape=[ksize[-1]]) - variable_summaries(b) + if verbose : variable_summaries(b) if dilate>1: if bias_term: conv_out = b + tf.nn.atrous_conv2d(indata,W,rate = dilate,padding=padding,name=name) @@ -33,8 +34,8 @@ def conv_layer(indata,ksize,padding,training,name,dilate = 1,strides=[1,1,1,1],b # conv_out = batchnorm(conv_out,scope=scope,training = training) conv_out = simple_global_bn(conv_out,name = name+'_bn') if active: - with tf.variable_scope(name+'_relu'): - conv_out = tf.nn.relu(conv_out,name='relu') + with tf.variable_scope(name+'_swish'): + conv_out = conv_out*tf.nn.sigmoid(beta*conv_out,name='swish') return conv_out def batchnorm(inp,scope,training,decay = 0.99,epsilon = 1e-5): with tf.variable_scope(scope): @@ -86,19 +87,19 @@ def inception_layer(indata,training,times=16): conv0f = conv_layer(indata,ksize=[1,1,in_channel,times*2],padding = 'SAME',training = training,name = 'conv0f_1x1') conv1f = conv_layer(conv0f,ksize=[1,3,times*2,times*3],padding = 'SAME',training = training,name = 'conv1f_1x3_d3',dilate = 3) return(tf.concat([conv1a,conv0b,conv1c,conv1d,conv1e,conv1f],axis = -1,name = 'concat')) -def residual_layer(indata,out_channel,training,i_bn = False): +def residual_layer(indata,out_channel,training,i_bn = False,verbose=False): fea_shape = indata.get_shape().as_list() in_channel = fea_shape[-1] with tf.variable_scope('branch1'): - indata_cp = conv_layer(indata,ksize = [1,1,in_channel,out_channel],padding = 'SAME',training = training,name = 'conv1',BN = i_bn,active = False) + indata_cp = conv_layer(indata,ksize = [1,1,in_channel,out_channel],padding = 'SAME',training = training,name = 'conv1',BN = i_bn,active = False,verbose=verbose) with tf.variable_scope('branch2'): - conv_out1 = conv_layer(indata,ksize = [1,1,in_channel,out_channel],padding = 'SAME',training = training,name = 'conv2a',bias_term = False) - conv_out2 = conv_layer(conv_out1,ksize = [1,3,out_channel,out_channel],padding = 'SAME',training=training,name = 'conv2b',bias_term = False) - conv_out3 = conv_layer(conv_out2,ksize = [1,1,out_channel,out_channel],padding = 'SAME',training=training,name = 'conv2c',bias_term = False,active = False) + conv_out1 = conv_layer(indata,ksize = [1,1,in_channel,out_channel],padding = 'SAME',training = training,name = 'conv2a',bias_term = False,verbose=verbose) + conv_out2 = conv_layer(conv_out1,ksize = [1,3,out_channel,out_channel],padding = 'SAME',training=training,name = 'conv2b',bias_term = False,verbose=verbose) + conv_out3 = conv_layer(conv_out2,ksize = [1,1,out_channel,out_channel],padding = 'SAME',training=training,name = 'conv2c',bias_term = False,active = False,verbose=verbose) with tf.variable_scope('plus'): relu_out = tf.nn.relu(indata_cp+conv_out3,name = 'final_relu') return relu_out -def getcnnfeature(signal,training): +def getcnnfeature(signal,training,verbose=False): signal_shape = signal.get_shape().as_list() signal = tf.reshape(signal,[signal_shape[0],1,signal_shape[1],1]) print(signal.get_shape()) @@ -147,11 +148,11 @@ def getcnnfeature(signal,training): # Residual Layer x 5 with tf.variable_scope('res_layer1'): - res1 = residual_layer(signal,out_channel = 256,training = training,i_bn = True) + res1 = residual_layer(signal,out_channel = 256,training = training,i_bn = True,verbose=verbose) with tf.variable_scope('res_layer2'): - res2 = residual_layer(res1,out_channel = 256,training = training) + res2 = residual_layer(res1,out_channel = 256,training = training,verbose=verbose) with tf.variable_scope('res_layer3'): - res3 = residual_layer(res2,out_channel = 256,training = training) + res3 = residual_layer(res2,out_channel = 256,training = training,verbose=verbose) # with tf.variable_scope('res_layer4'): # res4 = residual_layer(res3,out_channel = 512,training = training) # with tf.variable_scope('res_layer5'): diff --git a/deepore_SRU+Swish/chiron/cnn.pyc b/deepore_SRU+Swish/chiron/cnn.pyc new file mode 100644 index 0000000..20ff497 Binary files /dev/null and b/deepore_SRU+Swish/chiron/cnn.pyc differ diff --git a/Chiron+weight_visual/chiron/entry.py b/deepore_SRU+Swish/chiron/entry.py similarity index 100% rename from Chiron+weight_visual/chiron/entry.py rename to deepore_SRU+Swish/chiron/entry.py diff --git a/Chiron+weight_visual/chiron/example_data/read1.fast5 b/deepore_SRU+Swish/chiron/example_data/read1.fast5 similarity index 100% rename from Chiron+weight_visual/chiron/example_data/read1.fast5 rename to deepore_SRU+Swish/chiron/example_data/read1.fast5 diff --git a/Chiron+weight_visual/chiron/example_data/read2.fast5 b/deepore_SRU+Swish/chiron/example_data/read2.fast5 similarity index 100% rename from Chiron+weight_visual/chiron/example_data/read2.fast5 rename to deepore_SRU+Swish/chiron/example_data/read2.fast5 diff --git a/Chiron+weight_visual/chiron/example_data/read3.fast5 b/deepore_SRU+Swish/chiron/example_data/read3.fast5 similarity index 100% rename from Chiron+weight_visual/chiron/example_data/read3.fast5 rename to deepore_SRU+Swish/chiron/example_data/read3.fast5 diff --git a/Chiron+weight_visual/chiron/example_data/read4.fast5 b/deepore_SRU+Swish/chiron/example_data/read4.fast5 similarity index 100% rename from Chiron+weight_visual/chiron/example_data/read4.fast5 rename to deepore_SRU+Swish/chiron/example_data/read4.fast5 diff --git a/Chiron+weight_visual/chiron/example_data/read5.fast5 b/deepore_SRU+Swish/chiron/example_data/read5.fast5 similarity index 100% rename from Chiron+weight_visual/chiron/example_data/read5.fast5 rename to deepore_SRU+Swish/chiron/example_data/read5.fast5 diff --git a/Chiron+weight_visual/chiron/export_test.py b/deepore_SRU+Swish/chiron/export_test.py similarity index 100% rename from Chiron+weight_visual/chiron/export_test.py rename to deepore_SRU+Swish/chiron/export_test.py diff --git a/Chiron+weight_visual/chiron/load_run_test.py b/deepore_SRU+Swish/chiron/load_run_test.py similarity index 100% rename from Chiron+weight_visual/chiron/load_run_test.py rename to deepore_SRU+Swish/chiron/load_run_test.py diff --git a/Chiron+weight_visual/chiron/model/DNA_default/checkpoint b/deepore_SRU+Swish/chiron/model/DNA_default/checkpoint similarity index 100% rename from Chiron+weight_visual/chiron/model/DNA_default/checkpoint rename to deepore_SRU+Swish/chiron/model/DNA_default/checkpoint diff --git a/Chiron+weight_visual/chiron/model/DNA_default/model.ckpt-85520.data-00000-of-00001 b/deepore_SRU+Swish/chiron/model/DNA_default/model.ckpt-85520.data-00000-of-00001 similarity index 100% rename from Chiron+weight_visual/chiron/model/DNA_default/model.ckpt-85520.data-00000-of-00001 rename to deepore_SRU+Swish/chiron/model/DNA_default/model.ckpt-85520.data-00000-of-00001 diff --git a/Chiron+weight_visual/chiron/model/DNA_default/model.ckpt-85520.index b/deepore_SRU+Swish/chiron/model/DNA_default/model.ckpt-85520.index similarity index 100% rename from Chiron+weight_visual/chiron/model/DNA_default/model.ckpt-85520.index rename to deepore_SRU+Swish/chiron/model/DNA_default/model.ckpt-85520.index diff --git a/Chiron+weight_visual/chiron/model/DNA_default/model.ckpt-85520.meta b/deepore_SRU+Swish/chiron/model/DNA_default/model.ckpt-85520.meta similarity index 100% rename from Chiron+weight_visual/chiron/model/DNA_default/model.ckpt-85520.meta rename to deepore_SRU+Swish/chiron/model/DNA_default/model.ckpt-85520.meta diff --git a/Chiron+weight_visual/chiron/rnn.py b/deepore_SRU+Swish/chiron/rnn.py similarity index 84% rename from Chiron+weight_visual/chiron/rnn.py rename to deepore_SRU+Swish/chiron/rnn.py index 4230408..61dce5b 100644 --- a/Chiron+weight_visual/chiron/rnn.py +++ b/deepore_SRU+Swish/chiron/rnn.py @@ -13,6 +13,7 @@ from utils.lstm import BNLSTMCell from tensorflow.contrib.rnn.python.ops.rnn import stack_bidirectional_dynamic_rnn from summary import variable_summaries +from sru import BNSRUCell def rnn_layers(x,seq_length,training,hidden_num=100,layer_num = 3,class_n = 5): cells_fw = list() @@ -39,19 +40,24 @@ def rnn_layers(x,seq_length,training,hidden_num=100,layer_num = 3,class_n = 5): lasth_output = tf.nn.bias_add(tf.reduce_sum(tf.multiply(lasth_rs,weight_out),axis = 2),biases_out,name = 'lasth_bias_add') lasth_output_rs = tf.reshape(lasth_output,[batch_size*max_time,hidden_num],name = 'lasto_rs') logits = tf.reshape(tf.nn.bias_add(tf.matmul(lasth_output_rs,weight_class),bias_class),[batch_size,max_time,class_n],name = "rnn_logits_rs") - variable_summaries(weight_class) - variable_summaries(biases_out) + if verbose: + variable_summaries(weight_class) + variable_summaries(biases_out) return logits -def rnn_layers_one_direction(x,seq_length,training,hidden_num=200,layer_num = 3,class_n = 5): +def rnn_layers_one_direction(x,seq_length,training,hidden_num=256,layer_num = 3,class_n = 5,verbose=False): cells = list() + #This is the thingwe need to change to add SRU for i in range(layer_num): - cell = BNLSTMCell(hidden_num,training) + cell = BNSRUCell(hidden_num,training) + #cell = BNLSTMCell(hidden_num,training) cells.append(cell) + # This stacks the rnn ontop of each other cell_wrap = tf.contrib.rnn.MultiRNNCell(cells) with tf.variable_scope('LSTM_rnn') as scope: lasth,_ = tf.nn.dynamic_rnn(cell_wrap,x,sequence_length = seq_length,dtype = tf.float32,scope = scope) #shape of lasth [batch_size,max_time,hidden_num*2] + # This is the fully connected layer batch_size = lasth.get_shape().as_list()[0] max_time = lasth.get_shape().as_list()[1] with tf.variable_scope('rnn_fnn_layer'): @@ -59,6 +65,7 @@ def rnn_layers_one_direction(x,seq_length,training,hidden_num=200,layer_num = 3, bias_class = tf.Variable(tf.zeros([class_n]),name = 'bias_class') lasth_rs = tf.reshape(lasth,[batch_size*max_time,hidden_num],name = 'lasth_rs') logits = tf.reshape(tf.nn.bias_add(tf.matmul(lasth_rs,weight_class),bias_class),[batch_size,max_time,class_n],name = "rnn_logits_rs") - variable_summaries(weight_class) - variable_summaries(biases_out) + if verbose : + variable_summaries(weight_class) + variable_summaries(bias_class) return logits diff --git a/deepore_SRU+Swish/chiron/rnn.pyc b/deepore_SRU+Swish/chiron/rnn.pyc new file mode 100644 index 0000000..2e9ab91 Binary files /dev/null and b/deepore_SRU+Swish/chiron/rnn.pyc differ diff --git a/deepore_SRU+Swish/chiron/sru.py b/deepore_SRU+Swish/chiron/sru.py new file mode 100644 index 0000000..3696ef0 --- /dev/null +++ b/deepore_SRU+Swish/chiron/sru.py @@ -0,0 +1,111 @@ +import tensorflow as tf +import collections + +from tensorflow.python.ops import variable_scope as vs +from tensorflow.contrib.rnn import RNNCell + +def batch_norm(x, name_scope, training, epsilon=1e-3, decay=0.99): + '''Assume 2d [batch, values] tensor''' + + with tf.variable_scope(name_scope): + size = x.get_shape().as_list()[1] + + scale = tf.get_variable('scale', shape = [size], initializer=tf.constant_initializer(0.1)) + offset = tf.get_variable('offset', shape = [size]) + + pop_mean = tf.get_variable('pop_mean', shape = [size], initializer=tf.zeros_initializer(), trainable=False) + pop_var = tf.get_variable('pop_var', shape = [size], initializer=tf.ones_initializer(), trainable=False) + batch_mean, batch_var = tf.nn.moments(x, [0]) + + train_mean_op = tf.assign(pop_mean, pop_mean * decay + batch_mean * (1 - decay)) + train_var_op = tf.assign(pop_var, pop_var * decay + batch_var * (1 - decay)) + + def batch_statistics(): + with tf.control_dependencies([train_mean_op, train_var_op]): + return tf.nn.batch_normalization(x, batch_mean, batch_var, offset, scale, epsilon) + + def population_statistics(): + return tf.nn.batch_normalization(x, pop_mean, pop_var, offset, scale, epsilon) + + return tf.cond(training, batch_statistics, population_statistics) + +class BNSRUCell(RNNCell): + def __init__(self, num_units, training,activation=tf.nn.tanh, state_is_tuple=False, reuse=None): + super(BNSRUCell, self).__init__(_reuse=reuse) + self.hidden_dim = num_units + self.state_is_tuple = state_is_tuple + self.g = activation + init_matrix = tf.orthogonal_initializer() + + self.Wr = tf.Variable(init_matrix([self.hidden_dim, self.hidden_dim])) + self.br = tf.Variable(self.init_matrix([self.hidden_dim])) + + self.Wf = tf.Variable(init_matrix([self.hidden_dim, self.hidden_dim])) + self.bf = tf.Variable(self.init_matrix([self.hidden_dim])) + + self.U = tf.Variable(init_matrix([self.hidden_dim, self.hidden_dim])) + self.training = training + + @property + def state_size(self): + return self.hidden_dim + + @property + def output_size(self): + return self.hidden_dim + + def __call__(self, inputs, state, scope=None): + with vs.variable_scope(scope or type(self).__name__): + if self.state_is_tuple: + (c_prev, h_prev) = state + else: + c_prev = state + # Forget Gate + f = tf.sigmoid( + tf.matmul(inputs, self.Wf) + self.bf + ) + + f = batch_norm(f,'f',self.training) + + # Reset Gate + r = tf.sigmoid( + tf.matmul(inputs, self.Wr) + self.br + ) + + r = batch_norm(r,'r',self.training) + + # Final Memory cell + c = f * c_prev + (1.0 - f) * tf.matmul(inputs, self.U) + + c = batch_norm(c,'c',self.training) + # Current Hidden state + current_hidden_state = r * self.g(c) + (1.0 - r) * inputs + if self.state_is_tuple: + return current_hidden_state, LSTMStateTuple(c, current_hidden_state) + else: + return current_hidden_state, c + + def init_matrix(self, shape): + return tf.random_normal(shape, stddev=0.1) + + +_LSTMStateTuple = collections.namedtuple("LSTMStateTuple", ("c", "h")) + + +class LSTMStateTuple(_LSTMStateTuple): + """Tuple used by LSTM Cells for `state_size`, `zero_state`, and output state. + + Stores two elements: `(c, h)`, in that order. + + Only used when `state_is_tuple=True`. + """ + __slots__ = () + + @property + def dtype(self): + (c, h) = self + if c.dtype != h.dtype: + raise TypeError("Inconsistent internal state: %s vs %s" % + (str(c.dtype), str(h.dtype))) + return c.dtype + diff --git a/deepore_SRU+Swish/chiron/sru.pyc b/deepore_SRU+Swish/chiron/sru.pyc new file mode 100644 index 0000000..bb71ee6 Binary files /dev/null and b/deepore_SRU+Swish/chiron/sru.pyc differ diff --git a/Chiron+weight_visual/chiron/summary.py b/deepore_SRU+Swish/chiron/summary.py similarity index 97% rename from Chiron+weight_visual/chiron/summary.py rename to deepore_SRU+Swish/chiron/summary.py index 56213fa..bc418d6 100644 --- a/Chiron+weight_visual/chiron/summary.py +++ b/deepore_SRU+Swish/chiron/summary.py @@ -1,3 +1,5 @@ +import tensorflow as tf + def variable_summaries(var): """Attach a lot of summaries to a Tensor (for TensorBoard visualization).""" with tf.name_scope('summaries'): diff --git a/deepore_SRU+Swish/chiron/summary.pyc b/deepore_SRU+Swish/chiron/summary.pyc new file mode 100644 index 0000000..ee4eb08 Binary files /dev/null and b/deepore_SRU+Swish/chiron/summary.pyc differ diff --git a/Chiron+weight_visual/chiron/train_test.py b/deepore_SRU+Swish/chiron/train_test.py similarity index 100% rename from Chiron+weight_visual/chiron/train_test.py rename to deepore_SRU+Swish/chiron/train_test.py diff --git a/Chiron+weight_visual/chiron/utils/__init__.py b/deepore_SRU+Swish/chiron/utils/__init__.py similarity index 100% rename from Chiron+weight_visual/chiron/utils/__init__.py rename to deepore_SRU+Swish/chiron/utils/__init__.py diff --git a/deepore_SRU+Swish/chiron/utils/__init__.pyc b/deepore_SRU+Swish/chiron/utils/__init__.pyc new file mode 100644 index 0000000..9d942ff Binary files /dev/null and b/deepore_SRU+Swish/chiron/utils/__init__.pyc differ diff --git a/Chiron+weight_visual/chiron/utils/assess.sh b/deepore_SRU+Swish/chiron/utils/assess.sh similarity index 100% rename from Chiron+weight_visual/chiron/utils/assess.sh rename to deepore_SRU+Swish/chiron/utils/assess.sh diff --git a/Chiron+weight_visual/chiron/utils/batch_assess.sh b/deepore_SRU+Swish/chiron/utils/batch_assess.sh similarity index 100% rename from Chiron+weight_visual/chiron/utils/batch_assess.sh rename to deepore_SRU+Swish/chiron/utils/batch_assess.sh diff --git a/Chiron+weight_visual/chiron/utils/cmle_training_preprocess.pl b/deepore_SRU+Swish/chiron/utils/cmle_training_preprocess.pl similarity index 100% rename from Chiron+weight_visual/chiron/utils/cmle_training_preprocess.pl rename to deepore_SRU+Swish/chiron/utils/cmle_training_preprocess.pl diff --git a/Chiron+weight_visual/chiron/utils/easy_assembler.py b/deepore_SRU+Swish/chiron/utils/easy_assembler.py similarity index 100% rename from Chiron+weight_visual/chiron/utils/easy_assembler.py rename to deepore_SRU+Swish/chiron/utils/easy_assembler.py diff --git a/Chiron+weight_visual/chiron/utils/extract_sig_ref.py b/deepore_SRU+Swish/chiron/utils/extract_sig_ref.py similarity index 100% rename from Chiron+weight_visual/chiron/utils/extract_sig_ref.py rename to deepore_SRU+Swish/chiron/utils/extract_sig_ref.py diff --git a/Chiron+weight_visual/chiron/utils/labelop.py b/deepore_SRU+Swish/chiron/utils/labelop.py similarity index 97% rename from Chiron+weight_visual/chiron/utils/labelop.py rename to deepore_SRU+Swish/chiron/utils/labelop.py index 1669e85..db360af 100644 --- a/Chiron+weight_visual/chiron/utils/labelop.py +++ b/deepore_SRU+Swish/chiron/utils/labelop.py @@ -1,215 +1,215 @@ -import h5py -import numpy as np - -def get_label_segment(fast5_fn, basecall_group, basecall_subgroup): - try: - fast5_data = h5py.File(fast5_fn, 'r') - except IOError: - raise IOError, 'Error opening file. Likely a corrupted file.' - - #Get samping rate - try: - fast5_info = fast5_data['UniqueGlobalKey/channel_id'].attrs - sampling_rate = fast5_info['sampling_rate'].astype('int_') - except: - raise RuntimeError, ('Could not get channel info') - - # Read raw data - try: - raw_dat = fast5_data['/Raw/Reads/'].values()[0] - raw_attrs = raw_dat.attrs - except: - raise RuntimeError, ( - 'Raw data is not stored in Raw/Reads/Read_[read#] so ' + - 'new segments cannot be identified.') - raw_start_time = raw_attrs['start_time'] - - #Read segmented data - try: - segment_dat = fast5_data[ - '/Analyses/' + basecall_group + '/' + basecall_subgroup + '/Events'] - segment_attrs = dict(segment_dat.attrs.items()) - segment_dat = segment_dat.value - - total = len(segment_dat) - - # Process segment data - segment_starts = segment_dat['start'] * sampling_rate - raw_start_time - segment_lengths = segment_dat['length'] * sampling_rate - segment_means = segment_dat['mean'] - segment_stdv = segment_dat['stdv'] - - # create the label for segment event - segment_kmer = np.full(segment_starts.shape, '-', dtype='S5') - segment_move = np.zeros(segment_starts.shape) - segment_cstart = np.zeros(segment_starts.shape) - segment_clength = np.zeros(segment_starts.shape) - - segment_data = np.array( - zip(segment_means, segment_stdv, segment_starts, segment_lengths, segment_kmer, segment_move, segment_cstart, segment_clength), - dtype=[('mean','float64'),('stdv','float64'),('start', '= len(segment_data)): - break - - if segment_data[segment_index]['start'] >= my_end: - break - #End of while true - corr_index += 1 - - if corr_index >= len(corr_starts) - 2: - break - kmer = kmer[1:] + corr_bases[corr_index + 2] - - -# print first_segment_index -# print segment_index -# print corr_index - segment_data = segment_data[first_segment_index:segment_index] - return (segment_data, first_segment_index, segment_index, total) - -def get_label_raw(fast5_fn, basecall_group, basecall_subgroup): - ##Open file - try: - fast5_data = h5py.File(fast5_fn, 'r') - except IOError: - raise IOError, 'Error opening file. Likely a corrupted file.' - - #Get raw data - try: - raw_dat = fast5_data['/Raw/Reads/'].values()[0] - #raw_attrs = raw_dat.attrs - raw_dat = raw_dat['Signal'].value - except: - raise RuntimeError, ( - 'Raw data is not stored in Raw/Reads/Read_[read#] so ' + - 'new segments cannot be identified.') - - # Read corrected data - try: - corr_data = fast5_data['/Analyses/RawGenomeCorrected_000/' + basecall_subgroup + '/Events'] - corr_attrs = dict(corr_data.attrs.items()) - corr_data = corr_data .value - except: - raise RuntimeError, ( - 'Corrected data know found.') - - fast5_info = fast5_data['UniqueGlobalKey/channel_id'].attrs - #sampling_rate = fast5_info['sampling_rate'].astype('int_') - - #Reading extra information - corr_start_rel_to_raw = corr_attrs['read_start_rel_to_raw']# - - if any(len(vals) <= 1 for vals in ( - corr_data, raw_dat)): - raise NotImplementedError, ( - 'One or no segments or signal present in read.') - - event_starts = corr_data['start'] + corr_start_rel_to_raw - event_lengths = corr_data['length'] - event_bases = corr_data['base'] - - fast5_data.close() - label_data = np.array( - zip(event_starts, event_lengths, event_bases), - dtype=[('start', '= len(segment_data)): + break + + if segment_data[segment_index]['start'] >= my_end: + break + #End of while true + corr_index += 1 + + if corr_index >= len(corr_starts) - 2: + break + kmer = kmer[1:] + corr_bases[corr_index + 2] + + +# print first_segment_index +# print segment_index +# print corr_index + segment_data = segment_data[first_segment_index:segment_index] + return (segment_data, first_segment_index, segment_index, total) + +def get_label_raw(fast5_fn, basecall_group, basecall_subgroup): + ##Open file + try: + fast5_data = h5py.File(fast5_fn, 'r') + except IOError: + raise IOError, 'Error opening file. Likely a corrupted file.' + + #Get raw data + try: + raw_dat = fast5_data['/Raw/Reads/'].values()[0] + #raw_attrs = raw_dat.attrs + raw_dat = raw_dat['Signal'].value + except: + raise RuntimeError, ( + 'Raw data is not stored in Raw/Reads/Read_[read#] so ' + + 'new segments cannot be identified.') + + # Read corrected data + try: + corr_data = fast5_data['/Analyses/RawGenomeCorrected_000/' + basecall_subgroup + '/Events'] + corr_attrs = dict(corr_data.attrs.items()) + corr_data = corr_data .value + except: + raise RuntimeError, ( + 'Corrected data know found.') + + fast5_info = fast5_data['UniqueGlobalKey/channel_id'].attrs + #sampling_rate = fast5_info['sampling_rate'].astype('int_') + + #Reading extra information + corr_start_rel_to_raw = corr_attrs['read_start_rel_to_raw']# + + if any(len(vals) <= 1 for vals in ( + corr_data, raw_dat)): + raise NotImplementedError, ( + 'One or no segments or signal present in read.') + + event_starts = corr_data['start'] + corr_start_rel_to_raw + event_lengths = corr_data['length'] + event_bases = corr_data['base'] + + fast5_data.close() + label_data = np.array( + zip(event_starts, event_lengths, event_bases), + dtype=[('start', '