TTS_Ryuich_T2+WN.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "TTS-taco2+WN.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "syjMmwOEa-uk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Tacotron2: WaveNet-basd text-to-speech demo\n",
        "\n",
        "- Tacotron2 (mel-spectrogram prediction part): https://github.com/Rayhane-mamah/Tacotron-2\n",
        "- WaveNet: https://github.com/r9y9/wavenet_vocoder\n",
        "\n",
        "This is a proof of concept for Tacotron2 text-to-speech synthesis. Models used here were trained on [LJSpeech dataset](https://keithito.com/LJ-Speech-Dataset/).\n",
        "\n",
        "**Notice**: The waveform generation is super slow since it implements naive autoregressive generation. It doesn't use parallel generation method described in [Parallel WaveNet](https://arxiv.org/abs/1711.10433). \n",
        "\n",
        "**Estimated time to complete**:  2~3 hours "
      ]
    },
    {
      "metadata": {
        "id": "zRSFD8xfo3aN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "modified by ** [Hyungon Ryu](mailto://hryu@nvidia.com)** | Sr. Solution Architect  for GPU inferencing in Google COLAB\n",
        "- Estimated time to complete : 10 minutes to configure. it depends on network traffic. \n",
        " - framework\n",
        "   - install CUDA 9.0\n",
        "   - install tensorflow-gpu == 0.9.0\n",
        "   - install pytorch == 0.4.1\n",
        "  - git clone\n",
        "    - Tacotron2\n",
        "    - WaveNet\n",
        " \n",
        "- Generation time : 26 minutes for 1 sec sentence. \n",
        "  - Mel generation : 20 sec\n",
        "  - Wave Generation : 25 min."
      ]
    },
    {
      "metadata": {
        "id": "NIPhtfaQYsXh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "JooJmceaYsl6",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "m7R_1MpFc3Za",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Setup\n",
        "\n",
        "### Install dependencies"
      ]
    },
    {
      "metadata": {
        "id": "3Ib1B0xqvYqF",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "pip3 uninstall -y tensorflow tensorflow-gpu pytorch torch\n",
        "rm -rf cuda-repo*\n",
        "wget https://developer.nvidia.com/compute/cuda/9.0/Prod/local_installers/cuda-repo-ubuntu1704-9-0-local_9.0.176-1_amd64-deb\n",
        "apt-get install dirmngr\n",
        "dpkg -i cuda-repo-ubuntu1704-9-0-local_9.0.176-1_amd64-deb\n",
        "apt-key add /var/cuda-repo-9-0-local/7fa2af80.pub\n",
        "apt-get update\n",
        "apt-get install  -y --no-install-recommends  \\\n",
        " cuda-core-9-0 \\\n",
        " cuda-cublas-9-0 cuda-cublas-dev-9-0 cuda-cudart-9-0 cuda-cudart-dev-9-0 \\\n",
        " cuda-cufft-9-0 cuda-cufft-dev-9-0 cuda-curand-9-0 cuda-curand-dev-9-0 \\\n",
        " cuda-cusolver-9-0 cuda-cusolver-dev-9-0 cuda-cusparse-9-0 \\\n",
        " cuda-cusparse-dev-9-0 \\\n",
        " cuda-libraries-9-0 cuda-libraries-dev-9-0 \\\n",
        " cuda-misc-headers-9-0 cuda-npp-9-0 cuda-npp-dev-9-0 \\\n",
        " cuda-nvgraph-9-0 cuda-nvgraph-dev-9-0 cuda-nvml-dev-9-0 cuda-nvrtc-9-0 \\\n",
        " cuda-nvrtc-dev-9-0\n",
        "rm -rf cuda-repo*\n",
        "rm -rf wget-log*\n",
        "pip3 install -q tensorflow-gpu==1.9.0  \n",
        "pip3 install -q torch"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NlLC7Q7Us8go",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%time\n",
        "import os\n",
        "from os.path import exists, join, expanduser\n",
        "\n",
        "os.chdir(expanduser(\"~\"))\n",
        "\n",
        "wavenet_dir = \"wavenet_vocoder\"\n",
        "if not exists(wavenet_dir):\n",
        "  ! git clone https://github.com/r9y9/$wavenet_dir\n",
        "    \n",
        "taco2_dir = \"Tacotron-2\"\n",
        "if not exists(taco2_dir):\n",
        "  ! git clone https://github.com/r9y9/$taco2_dir\n",
        "  ! cd $taco2_dir && git checkout -B wavenet3 origin/wavenet3"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "KBFfji_Avluz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Install dependencies\n",
        "os.chdir(join(expanduser(\"~\"), taco2_dir))\n",
        "!pip install -q -r requirements.txt\n",
        "\n",
        "os.chdir(join(expanduser(\"~\"), wavenet_dir))\n",
        "!pip install -q -e '.[train]'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "15p8phXx6nxe",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import tensorflow\n",
        "print('for tacotron2 tf version:', tensorflow.__version__)\n",
        "print('for wavenet pyt version :', torch.__version__)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "S3NIMS82ANxU",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "config = tf.ConfigProto()\n",
        "config.gpu_options.allow_growth = True\n",
        "session = tf.Session(config=config )"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sxNvGKhEoj0R",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_fZo1X7ac_Tp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Download pretrained models\n",
        "\n",
        "#### Tacotron2 (mel-spectrogram prediction part)"
      ]
    },
    {
      "metadata": {
        "id": "Sau06KhizkoD",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%time\n",
        "os.chdir(join(expanduser(\"~\"), taco2_dir))\n",
        "! mkdir -p logs-Tacotron\n",
        "if not exists(\"logs-Tacotron/pretrained\"):\n",
        "  ! curl -O -L \"https://www.dropbox.com/s/vx7y4qqs732sqgg/pretrained.tar.gz\"\n",
        "  ! tar xzvf pretrained.tar.gz\n",
        "  ! mv pretrained logs-Tacotron"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "D4tWl_hfdXdh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### WaveNet"
      ]
    },
    {
      "metadata": {
        "id": "q2kwJ-t_ykXZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%time\n",
        "os.chdir(join(expanduser(\"~\"), wavenet_dir))\n",
        "wn_preset = \"20180510_mixture_lj_checkpoint_step000320000_ema.json\"\n",
        "wn_checkpoint_path = \"20180510_mixture_lj_checkpoint_step000320000_ema.pth\"\n",
        "\n",
        "if not exists(wn_preset):\n",
        "  !curl -O -L \"https://www.dropbox.com/s/0vsd7973w20eskz/20180510_mixture_lj_checkpoint_step000320000_ema.json\"\n",
        "if not exists(wn_checkpoint_path):\n",
        "  !curl -O -L \"https://www.dropbox.com/s/zdbfprugbagfp2w/20180510_mixture_lj_checkpoint_step000320000_ema.pth\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "km1SAASEcIL6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Input texts to be synthesized\n",
        "\n",
        "Choose your favorite sentences :)"
      ]
    },
    {
      "metadata": {
        "id": "4LeTMHHFdcmS",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "os.chdir(join(expanduser(\"~\"), taco2_dir))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "tU1lz6PcbXut",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "cat << EOS > text_list.txt\n",
        "This will was a deliberate forgery.\n",
        "EOS\n",
        "\n",
        "cat text_list.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "K9akhzMhbWe0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Mel-spectrogram prediction by Tacoron2"
      ]
    },
    {
      "metadata": {
        "id": "CDi0Wn_QBYt5",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Remove old files if exist\n",
        "!rm -rf tacotron_output"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0n4h5aa51dHS",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%time\n",
        "!python synthesize.py --model='Tacotron' --mode='eval' \\\n",
        "  --hparams='symmetric_mels=False,max_abs_value=4.0,power=1.1,outputs_per_step=1' \\\n",
        "  --text_list=./text_list.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "tzbRL9KQA7tm",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!ls -alh ../Tacotron-2/tacotron_output/eval\n",
        "!ls -alh ../Tacotron-2/tacotron_output/logs-eval/plots\n",
        "!ls -alh ../Tacotron-2/tacotron_output/logs-eval/wavs"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "vkAA-n3mA_nC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# copy hidden files to COLAB ~\n",
        "!cp ../Tacotron-2/tacotron_output//eval/speech-mel-00001.npy /content/.\n",
        "!cp ../Tacotron-2/tacotron_output/logs-eval/wavs/* /content/.\n",
        "!cp ../Tacotron-2/tacotron_output/logs-eval/plots/* /content/."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FF1mh1Jvdp0a",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Waveform synthesis by WaveNet"
      ]
    },
    {
      "metadata": {
        "id": "rY_MfE0m8Ese",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%time\n",
        "%%bash\n",
        "pip install ipykernel"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Ys4e9N3Ox3l3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import librosa.display\n",
        "import IPython\n",
        "from IPython.display import Audio\n",
        "import numpy as np\n",
        "import torch"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "vTmp0T0G3lU0",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%time\n",
        "os.chdir(join(expanduser(\"~\"), wavenet_dir))\n",
        "\n",
        "# Setup WaveNet vocoder hparams\n",
        "from hparams import hparams\n",
        "with open(wn_preset) as f:\n",
        "    hparams.parse_json(f.read())\n",
        "\n",
        "# Setup WaveNet vocoder\n",
        "from train import build_model\n",
        "from synthesis import wavegen\n",
        "import torch\n",
        "\n",
        "use_cuda = torch.cuda.is_available()\n",
        "device = torch.device(\"cuda\" if use_cuda else \"cpu\")\n",
        "\n",
        "model = build_model().to(device)\n",
        "print(\"Load checkpoint from {}\".format(wn_checkpoint_path))\n",
        "checkpoint = torch.load(wn_checkpoint_path)\n",
        "model.load_state_dict(checkpoint[\"state_dict\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "334X6oFK6Vf9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%%time\n",
        "from glob import glob\n",
        "from tqdm import tqdm\n",
        "\n",
        "with open(\"../Tacotron-2/tacotron_output/eval/map.txt\") as f:\n",
        "  maps = f.readlines()\n",
        "maps = list(map(lambda x:x[:-1].split(\"|\"), maps))\n",
        "# filter out invalid ones\n",
        "maps = list(filter(lambda x:len(x) == 2, maps))\n",
        "\n",
        "print(\"List of texts to be synthesized\")\n",
        "for idx, (text,_) in enumerate(maps):\n",
        "  print(idx, text)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "QiXg8KrTCGda",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "print(model)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "yaleFjoyiND_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Waveform generation\n",
        "\n",
        "**Note**: This will takes hours to finish depending on the number and lenght of texts. Try short sentences first if you would like to see samples quickly."
      ]
    },
    {
      "metadata": {
        "id": "j9BO7IES7Htp",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "waveforms = []\n",
        "\n",
        "for idx, (text, mel) in enumerate(maps):\n",
        "  print(\"\\n\", idx, text)\n",
        "  mel_path = join(\"../Tacotron-2\", mel)\n",
        "  c = np.load(mel_path)\n",
        "  if c.shape[1] != hparams.num_mels:\n",
        "    np.swapaxes(c, 0, 1)\n",
        "  # Range [0, 4] was used for training Tacotron2 but WaveNet vocoder assumes [0, 1]\n",
        "  c = np.interp(c, (0, 4), (0, 1))\n",
        " \n",
        "  # Generate wave\n",
        "  waveform = wavegen(model, c=c, g=None, fast=True, tqdm=tqdm)\n",
        "  # Audio\n",
        "  IPython.display.display(Audio(waveform, rate=hparams.sample_rate))\n",
        "  waveforms.append(waveform)\n",
        "  \n",
        "\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "hNG8oI4OiJkJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Summary: audio samples"
      ]
    },
    {
      "metadata": {
        "id": "OIyfhn0v9Ntg",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "for idx, (text, mel) in enumerate(maps):\n",
        "  print(idx, text)\n",
        "  IPython.display.display(Audio(waveforms[idx], rate=hparams.sample_rate))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "O0hc4ah-gMUa",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "For more information, please visit https://github.com/r9y9/wavenet_vocoder. More samples can  be  found at https://r9y9.github.io/wavenet_vocoder/. "
      ]
    }
  ]
}