Home

Welcome to the neurose wiki!

Weekly reports

1. Weekly report 1
2. Weekly report 2
3. Weekly report 3
4. Weekly report 4
5. Weekly report 5
6. Weekly report 6

Other documentation

• Specification
• Test coverage
• Testing document
• Implementation document
• Usage instructions

Useful notes

• Calculations
• Notes about training a neural network
• Notes about backpropagation
• Notes about forward pass with matrices

Ideas for the future

• Convolutional layer (ongoing)
• Batch normalisation layer
• Inverse dropout layer

An example of implementing a droput layer:

# from Standford CS231 materials
p = 0.5 # probability of keeping a unit active. higher = less dropout

def train_step(X):
  # forward pass for example 3-layer neural network
  H1 = np.maximum(0, np.dot(W1, X) + b1)
  U1 = (np.random.rand(*H1.shape) < p) / p # first dropout mask. Notice /p!
  H1 *= U1 # drop!
  H2 = np.maximum(0, np.dot(W2, H1) + b2)
  U2 = (np.random.rand(*H2.shape) < p) / p # second dropout mask. Notice /p!
  H2 *= U2 # drop!
  out = np.dot(W3, H2) + b3
  
  # backward pass: compute gradients... (not shown)
  # perform parameter update... (not shown)
  
def predict(X):
  # ensembled forward pass
  H1 = np.maximum(0, np.dot(W1, X) + b1) # no scaling necessary
  H2 = np.maximum(0, np.dot(W2, H1) + b2)
  out = np.dot(W3, H2) + b3