IndRnnLayer.md

CIndRnnLayer Class

• CIndRnnLayer Class
  • Settings
    • Hidden layer size
    • Dropout rate
    • Reverse sequences
    • Activation
  • Trainable parameters
    • Weight matrix W
    • Weight vector U
    • Free terms B
  • Inputs
  • Outputs

This class implements IndRNN from this article.

It's a simple recurrent unit with the following formula:

    Y_t = activation( W * X_t + B + U * Y_t-1 )

where:

• W and B are weight matrix and free terms of the fully-connected layer respectively (W * X_t means matrix-by-vector multiplication)
• U is a recurrent weights vector (U * Y_t-1 means element-wise multiplication of 2 vectors of same length)
• activation is an activation function (sigmoid or ReLU)

Settings

Hidden layer size

void SetHiddenSize( int size );

Sets the hidden layer size. It affects the output size.

Dropout rate

void SetDropoutRate( float dropoutRate );

Sets the rate of dropout, applied to both input (X_t) and recurrent part (Y_t-1).

Reverse sequences

void SetReverseSequence( bool reverse );

Elements of the sequences are processed in reversed order if this flag is set.

Activation

void SetActivation( TActivationFunction activation );

Sets the activation function used in recurrent part. AF_Sigmoid by default.

Trainable parameters

Weight matrix W

CPtr<CDnnBlob> GetInputWeights() const;

The weight matrix W from the formula.

It has the following shape:

• BatchLength * BatchWidth * ListSize is equal to GetHiddenSize()
• Height * Width * Depth * Channels is equal to the product of the same dimensions of the input.

Weight vector U

CPtr<CDnnBlob> GetRecurrentWeights() const;

The weight vector U from the formula. It's represented by a blob of the total size GetHiddenSize().

Free terms B

CPtr<CDnnBlob> GetBias() const

The free terms B from the formula. It's represented by a blob of the total size GetHiddenSize().

Inputs

The single input of this layer accepts the set of vector sequences of the following shape:

• BatchLength - the length of one vector sequence.
• BatchWidth * ListSize - the number of vector sequences in the input set.
• Height * Width * Depth * Channels - the size of each vector in the sequence.

Outputs

The single output returns a blob of the following size:

• BatchLength, BatchWidth, and ListSize are equal to the same sizes of the first input.
• Height, Width, and Depth equal 1.
• Channels equals GetHiddenSize().