research/Generating Music with Keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How To Generate Music Using a LSTM Neural Network In Keras\n",
    "\n",
    "- RNN: Recurrent Neural Network, uses sequential information to make decisions\n",
    "- LSTM: Long Short Term Memory network, useful for recognizing and encoding problems requiring long-term memory\n",
    "- MUSIC21: Python toolkit, useful for generating and studying music & processing MIDI files. Will use for transfering output of neural network to musical notation\n",
    "- Keras: High-level neural network Application Programming Interface (API) that enables utilization of Tensorflow (open source machine learning network)\n",
    "\n",
    "#### Training:\n",
    "\n",
    "- Any set single-instrument MIDI files can be inputted\n",
    "- MIDI files use chords and notes as object types, which have properties pitch (A-G), octave, and offset\n",
    "- Neural network will predict which note or chord will be played next (using a prediction array)\n",
    "- For offest, the example has each note 0.5 after the previous, so did not include offsets in the prediction array\n",
    "\n",
    "#### Data Management:\n",
    "\n",
    "- Load the data from the midi files, convert them to code format\n",
    "- Make sure the files contain only one instrument\n",
    "- To the prediction array, append the pitch ID of each note, and append chords in the format \"a.b.c.\" where a, b, c are the notes in the chord\n",
    "- Create a mapping function to convert the string items to numerical data (which computers handle easier)\n",
    "- Create input sequences to train the network on, and their respective output sequences. Experiment to see what differences occur with different lengths of input sequence\n",
    "\n",
    "#### Designing the Model:\n",
    "\n",
    "- LSTM Layers- take sequences as input, outputs sequences or matrices\n",
    "- Dropout Layers- prevent overfitting during training by using zeros as a fraction of the inputs each time\n",
    "- Dense Layers- each input node is connected to each output node\n",
    "- Activation Layer- determines which activation function will be used to calculate output\n",
    "\n",
    "- parameter = # of nodes layer should have for LSTM, dropout, activation layers\n",
    "- model.fit() function in Keras does the training of the network (first parameter = list of inputs we created, second parameter = list of corresponding outputs)\n",
    "- article example trains the network through 200 iterations, saves data after each iteration using Model Checkpoints which save the node weights to a file\n",
    "\n",
    "#### Generating Music:\n",
    "\n",
    "- Load the weights that were saved during training into the file (model.load_weights())\n",
    "- Can use a random index point (which contains a sequence) or input a command to choose the starting point of the generated music\n",
    "- Create a mapping function to decode the output of the files into readable format (notes)\n",
    "- After the iitial sequence input, remove the first note of the sequence & put it through the network again to generate the next note\n",
    "- Chooses note with highest index value as next in sequence, and collect all outputs into single array\n",
    "- Once done, decode the output array into 'note' and 'chord' type objects (where chords contain notes), which a 0.5 offset between each\n",
    "- Create Music21 'stream' using this array, then use Music21 'write' to create a MIDI file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why an LSTM Over Other NN's?\n",
    "\n",
    "- LSTM allows RNN to remember the inputs they were given for a much longer time\n",
    "- LSTM can read, write, and delete things from, its memory\n",
    "- Learns over time (through training) which information is important enough to keep or not (input gate = remember, forget gate = delete, output get = use for deciding the current output)\n",
    "- \"keeps training relatively short and accuracy high\" \"keeps gradients steep\" (what does this mean??)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Relation to Our Project/ My Role\n",
    "\n",
    "- We can use similar methods of encoding notes/chords/MIDI files to generate our output\n",
    "- We will have to create a prediction array out of songs (will need to compress into single-instrument MIDI files)\n",
    "- Difference: our input will be in the form of images\n",
    "- My role may involve creating the mapping functions to encode/decode the MIDI files to be put into the NN (data management)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}