Lesson4_AWS_Big_Data_Processing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Lesson4-AWS-Big-Data-Processing.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/paiml/awsbigdata/blob/master/Lesson4_AWS_Big_Data_Processing.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wy6nhdcaVsCH",
        "colab_type": "text"
      },
      "source": [
        "# Lesson 4 Processing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c_Id55m6Jsbu",
        "colab_type": "text"
      },
      "source": [
        "## Pragmatic AI Labs\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e5p96AqpSDZa",
        "colab_type": "text"
      },
      "source": [
        "![alt text](https://paiml.com/images/logo_with_slogan_white_background.png)\n",
        "\n",
        "This notebook was produced by [Pragmatic AI Labs](https://paiml.com/).  You can continue learning about these topics by:\n",
        "\n",
        "*   Buying a copy of [Pragmatic AI: An Introduction to Cloud-Based Machine Learning](http://www.informit.com/store/pragmatic-ai-an-introduction-to-cloud-based-machine-9780134863917)\n",
        "*   Reading an online copy of [Pragmatic AI:Pragmatic AI: An Introduction to Cloud-Based Machine Learning](https://www.safaribooksonline.com/library/view/pragmatic-ai-an/9780134863924/)\n",
        "*  Watching video [Essential Machine Learning and AI with Python and Jupyter Notebook-Video-SafariOnline](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118) on Safari Books Online.\n",
        "* Watching video [AWS Certified Machine Learning-Speciality](https://learning.oreilly.com/videos/aws-certified-machine/9780135556597)\n",
        "* Purchasing video [Essential Machine Learning and AI with Python and Jupyter Notebook- Purchase Video](http://www.informit.com/store/essential-machine-learning-and-ai-with-python-and-jupyter-9780135261095)\n",
        "*   Viewing more content at [noahgift.com](https://noahgift.com/)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IdrxEbJvoqQN",
        "colab_type": "text"
      },
      "source": [
        "## 4.1 Identify the appropriate data processing technology for a given scenario\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CyW2ZhrxowNv",
        "colab_type": "text"
      },
      "source": [
        "### Big Data Processing Technologies\n",
        "\n",
        "![Big Data Processing](https://user-images.githubusercontent.com/58792/54287663-54e6c700-4563-11e9-9f60-441302bbc929.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HiWIU44fjyfZ",
        "colab_type": "text"
      },
      "source": [
        "### Real World Data Engineering\n",
        "\n",
        "![Data Engineering Example](https://user-images.githubusercontent.com/58792/54398331-d713e000-4676-11e9-9239-4ec35a250734.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k4V3F0bhowQ_",
        "colab_type": "text"
      },
      "source": [
        "## 4.2 Determine how to design and architect the data processing solution"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tWsoko-eo0g_",
        "colab_type": "text"
      },
      "source": [
        "### Overview\n",
        "\n",
        "![feedback loop](https://user-images.githubusercontent.com/58792/54288887-75178580-4565-11e9-8a5c-ba6929012222.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zty7ydffo0j6",
        "colab_type": "text"
      },
      "source": [
        "## 4.3 Determine the operational characteristics of the solution implemented"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RujUaBmBo0m-",
        "colab_type": "text"
      },
      "source": [
        "[Whiteboard]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-7R3yc49o0qB",
        "colab_type": "text"
      },
      "source": [
        "## 4.4 Understand Overview of AWS Processing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eHxMdw5Mvy1a",
        "colab_type": "text"
      },
      "source": [
        "### Overview\n",
        "![processing](https://user-images.githubusercontent.com/58792/54293313-fb839580-456c-11e9-848a-b47bdd309430.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vDPvXKnZo0wu",
        "colab_type": "text"
      },
      "source": [
        "## 4.5 Understand Elastic MapReduce (EMR)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mWLdPK77o0zw",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GbNGsd8ro02q",
        "colab_type": "text"
      },
      "source": [
        "## 4.6 Learn about Apache Hadoop - Intro"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XRlsLGLJo05e",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RnV_aoluo-Eu",
        "colab_type": "text"
      },
      "source": [
        "## 4.7 Apply EMR - Architecture"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lrCT2tkIo-H3",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XkORPI0eo-Ng",
        "colab_type": "text"
      },
      "source": [
        "## 4.8 Understand EMR - Operations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d1ACakrvo-Qu",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PVmn8qQeo-Tr",
        "colab_type": "text"
      },
      "source": [
        "## 4.9 Use EMR - Hive\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lsot1OAjo-XD",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i--k5E2oo-Z7",
        "colab_type": "text"
      },
      "source": [
        "## 4.10 Use EMR - Hbase\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Z7Bxbk0o-c6",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rhVUSKhWpOME",
        "colab_type": "text"
      },
      "source": [
        "## 4.11 Use EMR - Presto\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s6awB8dwpOPR",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c3KHfDNjpOSh",
        "colab_type": "text"
      },
      "source": [
        "## 4.12 Use EMR - Spark\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jdkqygRUpV-e",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DiPXUDZFpWBs",
        "colab_type": "text"
      },
      "source": [
        "## 4.13 Implement EMR - Storage & Compression\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q89o0eZypWEi",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "whVq6K7NpZQa",
        "colab_type": "text"
      },
      "source": [
        "## 4.14 Implement EMR - Lambda"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_FFOIA3NpbTR",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    }
  ]
}