1 - Introduction to sklearn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hi, sklearn!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading spam collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ham\tDid he say how fantastic I am by any chance, or anything need a bigger life lift as losing the will 2 live, do you think I would be the first person 2 die from N V Q? \r\n",
      "ham\tBlack shirt n blue jeans... I thk i c ü...\r\n",
      "ham\tIf e timing can, then i go w u lor...\r\n",
      "ham\tThey r giving a second chance to rahul dengra.\r\n",
      "ham\tI cant pick the phone right now. Pls send a message\r\n",
      "ham\tHaha good to hear, I'm officially paid and on the market for an 8th\r\n",
      "ham\tFfffffffff. Alright no way I can meet up with you sooner?\r\n",
      "ham\tBut i'm really really broke oh. No amount is too small even  &lt;#&gt; \r\n",
      "ham\tOnly 2% students solved this CAT question in 'xam... 5+3+2= &lt;#&gt;  9+2+4= &lt;#&gt;  8+6+3= &lt;#&gt;  then 7+2+5=????? Tell me the answer if u r brilliant...1thing.i got d answr.\r\n",
      "spam\t<Forwarded from 21870000>Hi - this is your Mailbox Messaging SMS alert. You have 4 messages. You have 21 matches. Please call back on 09056242159 to retrieve your messages and matches\r\n",
      "ham\tNo da:)he is stupid da..always sending like this:)don believe any of those message.pandy is a mental:)\r\n",
      "spam\tYou won't believe it but it's true. It's Incredible Txts! Reply G now to learn truly amazing things that will blow your mind. From O2FWD only 18p/txt\r\n",
      "ham\tIt's fine, imma get a drink or somethin. Want me to come find you?\r\n",
      "ham\tI can probably come by, everybody's done around  &lt;#&gt;  right?\r\n",
      "spam\tsexy sexy cum and text me im wet and warm and ready for some porn! u up for some fun? THIS MSG IS FREE RECD MSGS 150P INC VAT 2 CANCEL TEXT STOP\r\n",
      "ham\tK, jason says he's gonna be around so I'll be up there around  &lt;#&gt;\r\n",
      "ham\tMon okie lor... Haha, best is cheap n gd food la, ex oso okie... Depends on whether wana eat western or chinese food... Den which u prefer... \r\n",
      "ham\tThats cool! Sometimes slow and gentle. Sonetimes rough and hard :)\r\n",
      "ham\tCan you do online transaction?\r\n",
      "ham\tHows that watch resizing\r\n",
      "ham\tYes princess! I want to catch you with my big strong hands...\r\n",
      "ham\tHa ha cool cool chikku chikku:-):-DB-)\r\n",
      "ham\tYar... I tot u knew dis would happen long ago already.\r\n",
      "ham\tIf you r @ home then come down within 5 min\r\n",
      "ham\tHaha get used to driving to usf man, I know a lot of stoners\r\n",
      "ham\tHe is impossible to argue with and he always treats me like his sub, like he never released me ... Which he did and I will remind him of that if necessary\r\n",
      "ham\tShuhui say change 2 suntec steamboat? U noe where? Where r u now?\r\n",
      "ham\tIn other news after hassling me to get him weed for a week andres has no money. HAUGHAIGHGTUJHYGUJ\r\n",
      "ham\tI've sent ü my part..\r\n",
      "ham\tI'll text now! All creepy like so he won't think that we forgot\r\n",
      "ham\tWant to send me a virtual hug?... I need one\r\n",
      "ham\tHow will I creep on you now? ;_;\r\n",
      "ham\tHe said that he had a right giggle when he saw u again! You would possibly be the first person2die from NVQ, but think how much you could for! \r\n",
      "ham\tI ask if u meeting da ge tmr nite...\r\n",
      "spam\tFREE camera phones with linerental from 4.49/month with 750 cross ntwk mins. 1/2 price txt bundle deals also avble. Call 08001950382 or call2optout/J MF\r\n",
      "ham\tIs that on the telly? No its Brdget Jones!\r\n",
      "ham\tHey darlin.. i can pick u up at college if u tell me wen & where 2 mt.. love Pete xx\r\n",
      "ham\taathi..where are you dear..\r\n",
      "ham\tsoon you will have the real thing princess! Do i make you wet? :)\r\n",
      "ham\tIf you can make it any time tonight or whenever you can it's cool, just text me whenever you're around\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 40 ./data/1-sms-spam-train.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import codecs\n",
    "\n",
    "with codecs.open('./data/1-sms-spam-train.txt') as f:\n",
    "    labels, messages = zip(*[line.split('\\t') for line in f.readlines()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### read test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with codecs.open('./data/1-sms-spam-test.txt') as f:\n",
    "    kaggle_test_messages = f.readlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### prepare solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas\n",
    "from IPython.display import FileLink\n",
    "\n",
    "def create_solution(predictions, filename='1-sms-spam-predictions.csv'):\n",
    "    result = pandas.DataFrame({'Id': numpy.arange(len(predictions)), 'Label': predictions})\n",
    "    result.to_csv('data/{}'.format(filename), index=False)\n",
    "    return FileLink('data/{}'.format(filename))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def compute_data_expressions(messages):\n",
    "    features = []\n",
    "    # length of each string\n",
    "    features.append(map(len, messages))\n",
    "    \n",
    "    # number of letters, digits, spaces = words\n",
    "    for pattern in [str.isalpha, str.isdigit, str.isspace]:\n",
    "        features.append(map(lambda message: sum(map(pattern, message)), messages))\n",
    "        \n",
    "    features = numpy.array(features).T\n",
    "    return features\n",
    "\n",
    "features = compute_data_expressions(messages)\n",
    "kaggle_test_features = compute_data_expressions(kaggle_test_messages)\n",
    "\n",
    "answers = numpy.array(labels) == 'spam' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[168, 124,   2,  39],\n",
       "       [ 44,  26,   0,  10],\n",
       "       [ 38,  24,   0,  10],\n",
       "       ..., \n",
       "       [ 31,  22,   0,   6],\n",
       "       [175, 119,  21,  28],\n",
       "       [ 25,  20,   0,   5]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.997237808402064"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "# area under the roc curve\n",
    "from sklearn.metrics import roc_auc_score\n",
    "knn_clf = KNeighborsClassifier(n_neighbors=1)\n",
    "knn_clf.fit(features, answers)\n",
    "roc_auc_score(answers, knn_clf.predict_proba(features)[:, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<a href='data/1-sms-spam-predictions.csv' target='_blank'>data/1-sms-spam-predictions.csv</a><br>"
      ],
      "text/plain": [
       "/Users/axelr/Documents/Яндекс.Диск/MLatImperial2016/data/1-sms-spam-predictions.csv"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "create_solution(knn_clf.predict_proba(kaggle_test_features)[:, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "trainX, testX, trainY, testY = train_test_split(features, answers, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Knn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test 0.935098650052\n"
     ]
    }
   ],
   "source": [
    "knn_clf = KNeighborsClassifier(n_neighbors=1)\n",
    "knn_clf.fit(trainX, trainY)\n",
    "print 'test', roc_auc_score(testY, knn_clf.predict_proba(testX)[:, 1])\n",
    "# print 'train', roc_auc_score(trainY, knn_clf.predict_proba(trainX)[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding optimal number of neighbours:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 0.935098650052\n",
      "2 0.953595534787\n",
      "4 0.967989211953\n",
      "8 0.968775239414\n",
      "16 0.976476866274\n",
      "32 0.981092073382\n",
      "64 0.974277431637\n"
     ]
    }
   ],
   "source": [
    "for n_neighbors in [1, 2, 4, 8, 16, 32, 64]:\n",
    "    knn_clf = KNeighborsClassifier(n_neighbors=n_neighbors)\n",
    "    knn_clf.fit(trainX, trainY)\n",
    "    print n_neighbors, roc_auc_score(testY, knn_clf.predict_proba(testX)[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### what happens if the metric is changed?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.983277085497\n",
      "0.989762131544\n"
     ]
    }
   ],
   "source": [
    "knn_clf = KNeighborsClassifier(metric='canberra', n_neighbors=20)\n",
    "knn_clf.fit(trainX, trainY)\n",
    "print roc_auc_score(testY, knn_clf.predict_proba(testX)[:, 1])\n",
    "print roc_auc_score(trainY, knn_clf.predict_proba(trainX)[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bag of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vectorizer = CountVectorizer()\n",
    "vectorizer.fit(messages)\n",
    "counts = vectorizer.transform(messages).toarray()\n",
    "test_counts = vectorizer.transform(kaggle_test_messages).toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3000, 6294)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counts.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# vocabulary is dictionary which keeps correspondence between columns and words\n",
    "# vectorizer.vocabulary_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "trainX, testX, trainY, testY = train_test_split(counts, answers, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### gaussian"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.88849948078920049"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "nb_clf = GaussianNB()\n",
    "nb_clf.fit(trainX, trainY)\n",
    "roc_auc_score(testY, nb_clf.predict_proba(testX)[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### multinomial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.97836621668397372"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "nb_clf = MultinomialNB()\n",
    "nb_clf.fit(trainX, trainY)\n",
    "roc_auc_score(testY, nb_clf.predict_proba(testX)[:, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2250, 6294)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainX.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear regression + Ridge regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.989976347064\n",
      "1.0\n"
     ]
    }
   ],
   "source": [
    "ridge_clf = Ridge()\n",
    "ridge_clf.fit(trainX, trainY)\n",
    "print roc_auc_score(testY, ridge_clf.predict(testX))\n",
    "print roc_auc_score(trainY, ridge_clf.predict(trainX))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Exercise #0.** Play with regularization parameter of RidgeRegression, see how it affects quality on train and test.\n",
    "Check quality of best model by submitting to kaggle.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise #1.** Let's write the correspondence between columns and words (done below). Which words are most popular?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dictionary = numpy.empty(len(vectorizer.vocabulary_), dtype='O')\n",
    "for word, index in vectorizer.vocabulary_.iteritems():\n",
    "    dictionary[index] = word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([u'00', u'000', u'000pes', ..., u'zyada', u'\\xe8n', u'\\xfa1'], dtype=object)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Exercise #2. ** By analyzing coefficients in `ridge_clf.coef_`, determine which words have the highest impact on decision (= have the largest modulus of `coef_`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Exercise #3. **  Does combining features and counts improve quality? Use `numpy.hstack` to concatenate arrays.\n",
    "Explain the result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Exercise #4.** Print examples on which your classifier makes mistakes (both false positive and false negative).\n",
    "\n",
    "This is important step to understand what can be done to improve the classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Exercise #5. (optional, just for fun)**  write a spam SMS, which is not caught by your best model. \n",
    "Something like \"Send sms YES to 091231323 to activate amazing spam filter, FREE for two weeks, then 20p/day. Txt now!\".\n",
    "\n",
    "Use your knowledge about the structure of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Major Goal (not in the homework). ** Provide best classification model for the problem. \n",
    "\n",
    "You can start with computing new features:\n",
    "1. Computing occurences of symbols\n",
    "2. Ignoring the words with digits, dots, etc.\n",
    "3. Detect links, phones in text\n",
    "\n",
    "Or start with changing parameters of classifiers. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}