WaveNet_text_to_speech.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Tacotron2 and WaveNet  text-to-speech demo.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "[View in Colaboratory](https://colab.research.google.com/github/swayam01/MUSTani-Robot/blob/master/WaveNet_text_to_speech.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "syjMmwOEa-uk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Tacotron2: WaveNet-basd text-to-speech demo\n",
        "\n",
        "- Tacotron2 (mel-spectrogram prediction part): https://github.com/Rayhane-mamah/Tacotron-2\n",
        "- WaveNet: https://github.com/r9y9/wavenet_vocoder\n",
        "\n",
        "This is a proof of concept for Tacotron2 text-to-speech synthesis. Models used here were trained on [LJSpeech dataset](https://keithito.com/LJ-Speech-Dataset/).\n",
        "\n",
        "**Notice**: The waveform generation is super slow since it implements naive autoregressive generation. It doesn't use the method described in [Parallel WaveNet](https://arxiv.org/abs/1711.10433). \n",
        "\n",
        "**Estimated time to complete**: 2 ~ 3 hours."
      ]
    },
    {
      "metadata": {
        "id": "m7R_1MpFc3Za",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Setup\n",
        "\n",
        "### Install dependencies"
      ]
    },
    {
      "metadata": {
        "id": "NlLC7Q7Us8go",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import os\n",
        "from os.path import exists, join, expanduser\n",
        "\n",
        "wavenet_dir = \"wavenet_vocoder\"\n",
        "if not exists(wavenet_dir):\n",
        "  ! git clone https://github.com/r9y9/$wavenet_dir\n",
        "    \n",
        "taco2_dir = \"Tacotron-2\"\n",
        "if not exists(taco2_dir):\n",
        "  ! git clone https://github.com/r9y9/$taco2_dir\n",
        "  ! cd $taco2_dir && git checkout -B wavenet3 origin/wavenet3"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "KBFfji_Avluz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Install dependencies\n",
        "# ! pip install -q --upgrade \"tensorflow-gpu==1.6.0\"\n",
        "\n",
        "os.chdir(join(expanduser(\"~\"), taco2_dir))\n",
        "! pip install -q -r requirements.txt\n",
        "\n",
        "os.chdir(join(expanduser(\"~\"), wavenet_dir))\n",
        "! pip install -q -e '.[train]'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "15p8phXx6nxe",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import tensorflow\n",
        "tensorflow.__version__"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_fZo1X7ac_Tp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Download pretrained models\n",
        "\n",
        "#### Tacotron2 (mel-spectrogram prediction part)"
      ]
    },
    {
      "metadata": {
        "id": "Sau06KhizkoD",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "os.chdir(join(expanduser(\"~\"), taco2_dir))\n",
        "! mkdir -p logs-Tacotron\n",
        "if not exists(\"logs-Tacotron/pretrained\"):\n",
        "  ! curl -O -L \"https://www.dropbox.com/s/vx7y4qqs732sqgg/pretrained.tar.gz\"\n",
        "  ! tar xzvf pretrained.tar.gz\n",
        "  ! mv pretrained logs-Tacotron"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "D4tWl_hfdXdh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### WaveNet"
      ]
    },
    {
      "metadata": {
        "id": "q2kwJ-t_ykXZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "os.chdir(join(expanduser(\"~\"), wavenet_dir))\n",
        "wn_preset = \"20180510_mixture_lj_checkpoint_step000320000_ema.json\"\n",
        "wn_checkpoint_path = \"20180510_mixture_lj_checkpoint_step000320000_ema.pth\"\n",
        "\n",
        "if not exists(wn_preset):\n",
        "  !curl -O -L \"https://www.dropbox.com/s/0vsd7973w20eskz/20180510_mixture_lj_checkpoint_step000320000_ema.json\"\n",
        "if not exists(wn_checkpoint_path):\n",
        "  !curl -O -L \"https://www.dropbox.com/s/zdbfprugbagfp2w/20180510_mixture_lj_checkpoint_step000320000_ema.pth\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "km1SAASEcIL6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Input texts to be synthesized\n",
        "\n",
        "Choose your favorite sentences :)"
      ]
    },
    {
      "metadata": {
        "id": "4LeTMHHFdcmS",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "os.chdir(join(expanduser(\"~\"), taco2_dir))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "tU1lz6PcbXut",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "cat << EOS > text_list.txt\n",
        "Hello Mustans\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "82rVUIra0Ggd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "cat text_list.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "K9akhzMhbWe0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Mel-spectrogram prediction by Tacoron2"
      ]
    },
    {
      "metadata": {
        "id": "0n4h5aa51dHS",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Remove old files if exist\n",
        "! rm -rf tacotron_output\n",
        "! python synthesize.py --model='Tacotron' --mode='eval' \\\n",
        "  --hparams='symmetric_mels=False,max_abs_value=4.0,power=1.1,outputs_per_step=1' \\\n",
        "  --text_list=./text_list.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FF1mh1Jvdp0a",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Waveform synthesis by WaveNet"
      ]
    },
    {
      "metadata": {
        "id": "rY_MfE0m8Ese",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import librosa.display\n",
        "import IPython\n",
        "from IPython.display import Audio\n",
        "import numpy as np\n",
        "import torch"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "vTmp0T0G3lU0",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "os.chdir(join(expanduser(\"~\"), wavenet_dir))\n",
        "\n",
        "# Setup WaveNet vocoder hparams\n",
        "from hparams import hparams\n",
        "with open(wn_preset) as f:\n",
        "    hparams.parse_json(f.read())\n",
        "\n",
        "# Setup WaveNet vocoder\n",
        "from train import build_model\n",
        "from synthesis import wavegen\n",
        "import torch\n",
        "\n",
        "use_cuda = torch.cuda.is_available()\n",
        "device = torch.device(\"cuda\" if use_cuda else \"cpu\")\n",
        "\n",
        "model = build_model().to(device)\n",
        "\n",
        "print(\"Load checkpoint from {}\".format(wn_checkpoint_path))\n",
        "checkpoint = torch.load(wn_checkpoint_path)\n",
        "model.load_state_dict(checkpoint[\"state_dict\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "334X6oFK6Vf9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from glob import glob\n",
        "from tqdm import tqdm\n",
        "\n",
        "with open(\"../Tacotron-2/tacotron_output/eval/map.txt\") as f:\n",
        "  maps = f.readlines()\n",
        "maps = list(map(lambda x:x[:-1].split(\"|\"), maps))\n",
        "# filter out invalid ones\n",
        "maps = list(filter(lambda x:len(x) == 2, maps))\n",
        "\n",
        "print(\"List of texts to be synthesized\")\n",
        "for idx, (text,_) in enumerate(maps):\n",
        "  print(idx, text)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "yaleFjoyiND_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Waveform generation\n",
        "\n",
        "**Note**: This will takes hours to finish depending on the number and lenght of texts. Try short sentences first if you would like to see samples quickly."
      ]
    },
    {
      "metadata": {
        "id": "j9BO7IES7Htp",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "waveforms = []\n",
        "\n",
        "for idx, (text, mel) in enumerate(maps):\n",
        "  print(\"\\n\", idx, text)\n",
        "  mel_path = join(\"../Tacotron-2\", mel)\n",
        "  c = np.load(mel_path)\n",
        "  if c.shape[1] != hparams.num_mels:\n",
        "    np.swapaxes(c, 0, 1)\n",
        "  # Range [0, 4] was used for training Tacotron2 but WaveNet vocoder assumes [0, 1]\n",
        "  c = np.interp(c, (0, 4), (0, 1))\n",
        " \n",
        "  # Generate\n",
        "  waveform = wavegen(model, c=c, fast=True, tqdm=tqdm)\n",
        "  \n",
        "  waveforms.append(waveform)\n",
        "\n",
        "  # Audio\n",
        "  IPython.display.display(Audio(waveform, rate=hparams.sample_rate))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "nHWn3uYr1LaR",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import scipy\n",
        "scipy.io.wavfile.write(filename= 'Must.wav', rate=hparams.sample_rate, data=waveform)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "DQoSQ6kI2anp",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!pip install -U -q PyDrive\n",
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive\n",
        "from google.colab import auth\n",
        "from oauth2client.client import GoogleCredentials"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "EfIRo_gy5VvP",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "OWUFv2he5ZcY",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "uploaded = drive.CreateFile({'title': 'must.wav'})\n",
        "uploaded.SetContentFile('Must.wav')\n",
        "uploaded.Upload()\n",
        "print('Uploaded file with ID {}'.format(uploaded.get('id')))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "V0AFQFt95Zhk",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "hNG8oI4OiJkJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Summary: audio samples"
      ]
    },
    {
      "metadata": {
        "id": "OIyfhn0v9Ntg",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "for idx, (text, mel) in enumerate(maps):\n",
        "  print(idx, text)\n",
        "  IPython.display.display(Audio(waveforms[idx], rate=hparams.sample_rate))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "O0hc4ah-gMUa",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "For more information, please visit https://github.com/r9y9/wavenet_vocoder. More samples can  be  found at https://r9y9.github.io/wavenet_vocoder/. "
      ]
    }
  ]
}