data-analysis-xgboost-starter-0-35460-lb.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "9c355c2c-044d-2b87-e1cf-a6732fb9e802"
      },
      "source": [
        "# Identifying Duplicate Questions\n",
        "\n",
        "Welcome to the Quora Question Pairs competition! Here, our goal is to identify which questions asked on [Quora](https://www.quora.com/), a quasi-forum website with over 100 million visitors a month, are duplicates of questions that have already been asked. This could be useful, for example, to instantly provide answers to questions that have already been answered. We are tasked with predicting whether a pair of questions are duplicates or not, and submitting a binary prediction against the logloss metric.\n",
        "\n",
        "If you have any questions or want to discuss competitions/hardware/games/anything with other Kagglers, then join the KaggleNoobs Slack channel [here](https://goo.gl/gGWFXe). We also have regular AMAs with top Kagglers there.\n",
        "\n",
        "**And as always, if this helped you, some upvotes would be very much appreciated - that's where I get my motivation! :D**\n",
        "\n",
        "Let's dive right into the data!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d9f5b5bf-b8a8-a88a-a16c-7666be39bd7e"
      },
      "outputs": [],
      "source": [
        "import numpy as np # linear algebra\n",
        "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
        "import os\n",
        "import gc\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "%matplotlib inline\n",
        "\n",
        "pal = sns.color_palette()\n",
        "\n",
        "print('# File sizes')\n",
        "for f in os.listdir('../input'):\n",
        "    if 'zip' not in f:\n",
        "        print(f.ljust(30) + str(round(os.path.getsize('../input/' + f) / 1000000, 2)) + 'MB')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e9b03ade-149a-7d6a-d954-2db4ebb7806f"
      },
      "source": [
        "Looks like we are simply given two files this time round, one for the training set and one for the test set. They are relatively small compared to other recent competitions, weighing in at less than 400MB total.\n",
        "\n",
        "It's worth noting that there is a lot more testing data than training data. This could be a sign that some of the test data is dummy data designed to deter hand-labelling, and not included in the calculations, like we recently saw in the [DSTL competition](https://www.kaggle.com/c/dstl-satellite-imagery-feature-detection/leaderboard).\n",
        "\n",
        "Let's open up one of the datasets.\n",
        "\n",
        "## Training set"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0ac4cab6-214b-957b-bb2a-01f7e8d5ed2b"
      },
      "outputs": [],
      "source": [
        "df_train = pd.read_csv('../input/train.csv')\n",
        "df_train.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a90218ce-edfc-e4bb-92cd-e49d09c0615c"
      },
      "source": [
        "We are given a minimal number of data fields here, consisting of:\n",
        "\n",
        "**`id`:** Looks like a simple rowID    \n",
        "**`qid{1, 2}`:** The unique ID of each question in the pair    \n",
        "**`question{1, 2}`:** The actual textual contents of the questions.    \n",
        "**`is_duplicate`:** The **label** that we are trying to predict - whether the two questions are duplicates of each other."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "68e0a285-995b-ecb2-fde7-3486c982912b"
      },
      "outputs": [],
      "source": [
        "print('Total number of question pairs for training: {}'.format(len(df_train)))\n",
        "print('Duplicate pairs: {}%'.format(round(df_train['is_duplicate'].mean()*100, 2)))\n",
        "qids = pd.Series(df_train['qid1'].tolist() + df_train['qid2'].tolist())\n",
        "print('Total number of questions in the training data: {}'.format(len(\n",
        "    np.unique(qids))))\n",
        "print('Number of questions that appear multiple times: {}'.format(np.sum(qids.value_counts() > 1)))\n",
        "\n",
        "plt.figure(figsize=(12, 5))\n",
        "plt.hist(qids.value_counts(), bins=50)\n",
        "plt.yscale('log', nonposy='clip')\n",
        "plt.title('Log-Histogram of question appearance counts')\n",
        "plt.xlabel('Number of occurences of question')\n",
        "plt.ylabel('Number of questions')\n",
        "print()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f5144bde-9178-827c-6591-fc502323d5fa"
      },
      "source": [
        "In terms of questions, everything looks as I would expect here. Most questions only appear a few times, with very few questions appearing several times (and a few questions appearing many times). One question appears more than 160 times, but this is an outlier.\n",
        "\n",
        "We can see that we have a 37% positive class in this dataset. Since we are using the [LogLoss](https://www.kaggle.com/wiki/LogarithmicLoss) metric, and LogLoss looks at the actual predicts as opposed to the order of predictions, we should be able to get a decent score by creating a submission predicting the mean value of the label.\n",
        "\n",
        "## Test Submission"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "dc70e18f-f364-0518-1d85-f7409269c754"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import log_loss\n",
        "\n",
        "p = df_train['is_duplicate'].mean() # Our predicted probability\n",
        "print('Predicted score:', log_loss(df_train['is_duplicate'], np.zeros_like(df_train['is_duplicate']) + p))\n",
        "\n",
        "df_test = pd.read_csv('../input/test.csv')\n",
        "sub = pd.DataFrame({'test_id': df_test['test_id'], 'is_duplicate': p})\n",
        "sub.to_csv('naive_submission.csv', index=False)\n",
        "sub.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "3bdf7cc4-4625-b5e6-996c-ecb063981bf2"
      },
      "source": [
        "**0.55 on the leaderboard! Score!**\n",
        "\n",
        "However, not all is well. The discrepancy between our local score and the LB one indicates that the distribution of values on the leaderboard is very different to what we have here, which could cause problems with validation later on in the competition.\n",
        "\n",
        "According to this [excellent notebook by David Thaler](www.kaggle.com/davidthaler/quora-question-pairs/how-many-1-s-are-in-the-public-lb/notebook), using our score and submission we can calculate that we have about 16.5% positives in the test set. This is quite surprising to see, so it'll be something that will need to be taken into account in machine learning models.\n",
        "\n",
        "Next, I'll take a quick peek at the statistics of the test data before we look at the text itself.\n",
        "\n",
        "## Test Set"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ff61e6b2-08b2-1734-db03-2db3ff5ee268"
      },
      "outputs": [],
      "source": [
        "df_test = pd.read_csv('../input/test.csv')\n",
        "df_test.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "1477d9f9-7cbf-09f5-21af-6b1da642ee69"
      },
      "outputs": [],
      "source": [
        "print('Total number of question pairs for testing: {}'.format(len(df_test)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "39f2c240-caa5-675b-3e6e-f9195808b392"
      },
      "source": [
        "Nothing out of the ordinary here. We are once again given rowIDs and the textual data of the two questions. It is worth noting that we are not given question IDs here however for the two questions in the pair.\n",
        "\n",
        "It is also worth pointing out that the actual number of test rows are likely to be much lower than 2.3 million. According to the [data page](https://www.kaggle.com/c/quora-question-pairs/data), most of the rows in the test set are using auto-generated questions to pad out the dataset, and deter any hand-labelling. This means that the true number of rows that are scored could be very low.\n",
        "\n",
        "We can actually see in the head of the test data that some of the questions are obviously auto-generated, as we get delights such as \"How their can I start reading?\" and \"What foods fibre?\". Truly insightful questions.\n",
        "\n",
        "Now onto the good stuff - the text data!\n",
        "## Text analysis\n",
        "\n",
        "First off, some quick histograms to understand what we're looking at. **Most analysis here will be only on the training set, to avoid the auto-generated questions**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ba41ab36-b287-4c5d-4ec5-a2c28034baa7"
      },
      "outputs": [],
      "source": [
        "train_qs = pd.Series(df_train['question1'].tolist() + df_train['question2'].tolist()).astype(str)\n",
        "test_qs = pd.Series(df_test['question1'].tolist() + df_test['question2'].tolist()).astype(str)\n",
        "\n",
        "dist_train = train_qs.apply(len)\n",
        "dist_test = test_qs.apply(len)\n",
        "plt.figure(figsize=(15, 10))\n",
        "plt.hist(dist_train, bins=200, range=[0, 200], color=pal[2], normed=True, label='train')\n",
        "plt.hist(dist_test, bins=200, range=[0, 200], color=pal[1], normed=True, alpha=0.5, label='test')\n",
        "plt.title('Normalised histogram of character count in questions', fontsize=15)\n",
        "plt.legend()\n",
        "plt.xlabel('Number of characters', fontsize=15)\n",
        "plt.ylabel('Probability', fontsize=15)\n",
        "\n",
        "print('mean-train {:.2f} std-train {:.2f} mean-test {:.2f} std-test {:.2f} max-train {:.2f} max-test {:.2f}'.format(dist_train.mean(), \n",
        "                          dist_train.std(), dist_test.mean(), dist_test.std(), dist_train.max(), dist_test.max()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "0ed063c0-9d5c-bc09-9c48-22ff739859bd"
      },
      "source": [
        "We can see that most questions have anywhere from 15 to 150 characters in them. It seems that the test distribution is a little different from the train one, but not too much so (I can't tell if it is just the larger data reducing noise, but it also seems like the distribution is a lot smoother in the test set).\n",
        "\n",
        "One thing that catches my eye is the steep cut-off at 150 characters for the training set, for most questions, while the test set slowly decreases after 150. Could this be some sort of Quora question size limit?\n",
        "\n",
        "It's also worth noting that I've truncated this histogram at 200 characters, and that the max of the distribution is at just under 1200 characters for both sets - although samples with over 200 characters are very rare.\n",
        "\n",
        "Let's do the same for word count. I'll be using a naive method for splitting words (splitting on spaces instead of using a serious tokenizer), although this should still give us a good idea of the distribution."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2a86bc3f-1359-ef11-4411-ec5b54c6dca5"
      },
      "outputs": [],
      "source": [
        "dist_train = train_qs.apply(lambda x: len(x.split(' ')))\n",
        "dist_test = test_qs.apply(lambda x: len(x.split(' ')))\n",
        "\n",
        "plt.figure(figsize=(15, 10))\n",
        "plt.hist(dist_train, bins=50, range=[0, 50], color=pal[2], normed=True, label='train')\n",
        "plt.hist(dist_test, bins=50, range=[0, 50], color=pal[1], normed=True, alpha=0.5, label='test')\n",
        "plt.title('Normalised histogram of word count in questions', fontsize=15)\n",
        "plt.legend()\n",
        "plt.xlabel('Number of words', fontsize=15)\n",
        "plt.ylabel('Probability', fontsize=15)\n",
        "\n",
        "print('mean-train {:.2f} std-train {:.2f} mean-test {:.2f} std-test {:.2f} max-train {:.2f} max-test {:.2f}'.format(dist_train.mean(), \n",
        "                          dist_train.std(), dist_test.mean(), dist_test.std(), dist_train.max(), dist_test.max()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f7eb07e3-fe48-c4ca-359e-71e5c7a1a4b3"
      },
      "source": [
        "We see a similar distribution for word count, with most questions being about 10 words long. It looks to me like the distribution of the training set seems more \"pointy\", while on the test set it is wider. Nevertheless, they are quite similar.\n",
        "\n",
        "So what are the most common words? Let's take a look at a word cloud."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d8202c51-7d0e-bf00-591d-41ce37cef841"
      },
      "outputs": [],
      "source": [
        "from wordcloud import WordCloud\n",
        "cloud = WordCloud(width=1440, height=1080).generate(\" \".join(train_qs.astype(str)))\n",
        "plt.figure(figsize=(20, 15))\n",
        "plt.imshow(cloud)\n",
        "plt.axis('off')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "e486c424-b529-e366-b455-77ee0df3aa84"
      },
      "source": [
        "## Semantic Analysis\n",
        "\n",
        "Next, I will take a look at usage of different punctuation in questions - this may form a basis for some interesting features later on."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "371a1e30-e5cd-7a24-97d7-b505b5aefb78"
      },
      "outputs": [],
      "source": [
        "qmarks = np.mean(train_qs.apply(lambda x: '?' in x))\n",
        "math = np.mean(train_qs.apply(lambda x: '[math]' in x))\n",
        "fullstop = np.mean(train_qs.apply(lambda x: '.' in x))\n",
        "capital_first = np.mean(train_qs.apply(lambda x: x[0].isupper()))\n",
        "capitals = np.mean(train_qs.apply(lambda x: max([y.isupper() for y in x])))\n",
        "numbers = np.mean(train_qs.apply(lambda x: max([y.isdigit() for y in x])))\n",
        "\n",
        "print('Questions with question marks: {:.2f}%'.format(qmarks * 100))\n",
        "print('Questions with [math] tags: {:.2f}%'.format(math * 100))\n",
        "print('Questions with full stops: {:.2f}%'.format(fullstop * 100))\n",
        "print('Questions with capitalised first letters: {:.2f}%'.format(capital_first * 100))\n",
        "print('Questions with capital letters: {:.2f}%'.format(capitals * 100))\n",
        "print('Questions with numbers: {:.2f}%'.format(numbers * 100))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "656b602a-e321-6ee4-682a-e76959bccc19"
      },
      "source": [
        "# Initial Feature Analysis\n",
        "\n",
        "Before we create a model, we should take a look at how powerful some features are. I will start off with the word share feature from the benchmark model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ef13480a-2522-2d1f-cf8a-56670e0856ab"
      },
      "outputs": [],
      "source": [
        "from nltk.corpus import stopwords\n",
        "\n",
        "stops = set(stopwords.words(\"english\"))\n",
        "\n",
        "def word_match_share(row):\n",
        "    q1words = {}\n",
        "    q2words = {}\n",
        "    for word in str(row['question1']).lower().split():\n",
        "        if word not in stops:\n",
        "            q1words[word] = 1\n",
        "    for word in str(row['question2']).lower().split():\n",
        "        if word not in stops:\n",
        "            q2words[word] = 1\n",
        "    if len(q1words) == 0 or len(q2words) == 0:\n",
        "        # The computer-generated chaff includes a few questions that are nothing but stopwords\n",
        "        return 0\n",
        "    shared_words_in_q1 = [w for w in q1words.keys() if w in q2words]\n",
        "    shared_words_in_q2 = [w for w in q2words.keys() if w in q1words]\n",
        "    R = (len(shared_words_in_q1) + len(shared_words_in_q2))/(len(q1words) + len(q2words))\n",
        "    return R\n",
        "\n",
        "plt.figure(figsize=(15, 5))\n",
        "train_word_match = df_train.apply(word_match_share, axis=1, raw=True)\n",
        "plt.hist(train_word_match[df_train['is_duplicate'] == 0], bins=20, normed=True, label='Not Duplicate')\n",
        "plt.hist(train_word_match[df_train['is_duplicate'] == 1], bins=20, normed=True, alpha=0.7, label='Duplicate')\n",
        "plt.legend()\n",
        "plt.title('Label distribution over word_match_share', fontsize=15)\n",
        "plt.xlabel('word_match_share', fontsize=15)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "24ce20d7-2c37-9833-dfdb-b4aac6b37e96"
      },
      "source": [
        "Here we can see that this feature has quite a lot of predictive power, as it is good at separating the duplicate questions from the non-duplicate ones. Interestingly, it seems very good at identifying questions which are definitely different, but is not so great at finding questions which are definitely duplicates.\n",
        "\n",
        "## TF-IDF\n",
        "\n",
        "I'm now going to try to improve this feature, by using something called TF-IDF (term-frequency-inverse-document-frequency). This means that we weigh the terms by how **uncommon** they are, meaning that we care more about rare words existing in both questions than common one. This makes sense, as for example we care more about whether the word \"exercise\" appears in both than the word \"and\" - as uncommon words will be more indicative of the content.\n",
        "\n",
        "You may want to look into using sklearn's [TfidfVectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html) to compute weights if you are implementing this yourself, but as I am too lazy to read the documentation I will write a version in pure python with a few changes which I believe should help the score."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "99d8e8e1-e2fd-25c6-137d-4492b850e1b5"
      },
      "outputs": [],
      "source": [
        "from collections import Counter\n",
        "\n",
        "# If a word appears only once, we ignore it completely (likely a typo)\n",
        "# Epsilon defines a smoothing constant, which makes the effect of extremely rare words smaller\n",
        "def get_weight(count, eps=10000, min_count=2):\n",
        "    if count < min_count:\n",
        "        return 0\n",
        "    else:\n",
        "        return 1 / (count + eps)\n",
        "\n",
        "eps = 5000 \n",
        "words = (\" \".join(train_qs)).lower().split()\n",
        "counts = Counter(words)\n",
        "weights = {word: get_weight(count) for word, count in counts.items()}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "d791c6b1-9860-7c3b-2b96-0767e607d6dc"
      },
      "outputs": [],
      "source": [
        "print('Most common words and weights: \\n')\n",
        "print(sorted(weights.items(), key=lambda x: x[1] if x[1] > 0 else 9999)[:10])\n",
        "print('\\nLeast common words and weights: ')\n",
        "(sorted(weights.items(), key=lambda x: x[1], reverse=True)[:10])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "2c8f2e86-f96b-5d36-e2f1-5f4c4c903e26"
      },
      "outputs": [],
      "source": [
        "def tfidf_word_match_share(row):\n",
        "    q1words = {}\n",
        "    q2words = {}\n",
        "    for word in str(row['question1']).lower().split():\n",
        "        if word not in stops:\n",
        "            q1words[word] = 1\n",
        "    for word in str(row['question2']).lower().split():\n",
        "        if word not in stops:\n",
        "            q2words[word] = 1\n",
        "    if len(q1words) == 0 or len(q2words) == 0:\n",
        "        # The computer-generated chaff includes a few questions that are nothing but stopwords\n",
        "        return 0\n",
        "    \n",
        "    shared_weights = [weights.get(w, 0) for w in q1words.keys() if w in q2words] + [weights.get(w, 0) for w in q2words.keys() if w in q1words]\n",
        "    total_weights = [weights.get(w, 0) for w in q1words] + [weights.get(w, 0) for w in q2words]\n",
        "    \n",
        "    R = np.sum(shared_weights) / np.sum(total_weights)\n",
        "    return R"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "157fd891-0b63-22df-4129-e4267cb55ff0"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(15, 5))\n",
        "tfidf_train_word_match = df_train.apply(tfidf_word_match_share, axis=1, raw=True)\n",
        "plt.hist(tfidf_train_word_match[df_train['is_duplicate'] == 0].fillna(0), bins=20, normed=True, label='Not Duplicate')\n",
        "plt.hist(tfidf_train_word_match[df_train['is_duplicate'] == 1].fillna(0), bins=20, normed=True, alpha=0.7, label='Duplicate')\n",
        "plt.legend()\n",
        "plt.title('Label distribution over tfidf_word_match_share', fontsize=15)\n",
        "plt.xlabel('word_match_share', fontsize=15)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "f1814b89-b895-f442-0ba2-b4e2fc6b6288"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import roc_auc_score\n",
        "print('Original AUC:', roc_auc_score(df_train['is_duplicate'], train_word_match))\n",
        "print('   TFIDF AUC:', roc_auc_score(df_train['is_duplicate'], tfidf_train_word_match.fillna(0)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "28b46d9f-82fe-c37e-010c-a3e115687d41"
      },
      "source": [
        "So it looks like our TF-IDF actually got _worse_ in terms of overall AUC, which is a bit disappointing. (I am using the AUC metric since it is unaffected by scaling and similar, so it is a good metric for testing the predictive power of individual features.\n",
        "\n",
        "However, I still think that this feature should provide some extra information which is not provided by the original feature. Our next job is to combine these features and use it to make a prediction. For this, I will use our old friend XGBoost to make a classification model.\n",
        "\n",
        "## Rebalancing the Data\n",
        "However, before I do this, I would like to rebalance the data that XGBoost receives, since we have 37% positive class in our training data, and only 17% in the test data. By re-balancing the data so our training set has 17% positives, we can ensure that XGBoost outputs probabilities that will better match the data on the leaderboard, and should get a better score (since LogLoss looks at the probabilities themselves and not just the order of the predictions like AUC)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "0589da36-d092-c951-f3cd-68df3fd91c7a"
      },
      "outputs": [],
      "source": [
        "# First we create our training and testing data\n",
        "x_train = pd.DataFrame()\n",
        "x_test = pd.DataFrame()\n",
        "x_train['word_match'] = train_word_match\n",
        "x_train['tfidf_word_match'] = tfidf_train_word_match\n",
        "x_test['word_match'] = df_test.apply(word_match_share, axis=1, raw=True)\n",
        "x_test['tfidf_word_match'] = df_test.apply(tfidf_word_match_share, axis=1, raw=True)\n",
        "\n",
        "y_train = df_train['is_duplicate'].values"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "5ccae7db-46e1-cf74-fdd8-4d4758bf460f"
      },
      "outputs": [],
      "source": [
        "pos_train = x_train[y_train == 1]\n",
        "neg_train = x_train[y_train == 0]\n",
        "\n",
        "# Now we oversample the negative class\n",
        "# There is likely a much more elegant way to do this...\n",
        "p = 0.165\n",
        "scale = ((len(pos_train) / (len(pos_train) + len(neg_train))) / p) - 1\n",
        "while scale > 1:\n",
        "    neg_train = pd.concat([neg_train, neg_train])\n",
        "    scale -=1\n",
        "neg_train = pd.concat([neg_train, neg_train[:int(scale * len(neg_train))]])\n",
        "print(len(pos_train) / (len(pos_train) + len(neg_train)))\n",
        "\n",
        "x_train = pd.concat([pos_train, neg_train])\n",
        "y_train = (np.zeros(len(pos_train)) + 1).tolist() + np.zeros(len(neg_train)).tolist()\n",
        "del pos_train, neg_train"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "20684b12-fd83-bad6-b00b-77c51dd14c32"
      },
      "outputs": [],
      "source": [
        "# Finally, we split some of the data off for validation\n",
        "from sklearn.cross_validation import train_test_split\n",
        "\n",
        "x_train, x_valid, y_train, y_valid = train_test_split(x_train, y_train, test_size=0.2, random_state=4242)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "9df47145-da6a-ad51-7bfb-1b588c4bab06"
      },
      "source": [
        "## XGBoost\n",
        "\n",
        "Now we can finally run XGBoost on our data, in order to see the score on the leaderboard!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "a5f9f7fc-5ec4-fc9f-7fc0-608f5f25e7c6"
      },
      "outputs": [],
      "source": [
        "import xgboost as xgb\n",
        "\n",
        "# Set our parameters for xgboost\n",
        "params = {}\n",
        "params['objective'] = 'binary:logistic'\n",
        "params['eval_metric'] = 'logloss'\n",
        "params['eta'] = 0.02\n",
        "params['max_depth'] = 4\n",
        "\n",
        "d_train = xgb.DMatrix(x_train, label=y_train)\n",
        "d_valid = xgb.DMatrix(x_valid, label=y_valid)\n",
        "\n",
        "watchlist = [(d_train, 'train'), (d_valid, 'valid')]\n",
        "\n",
        "bst = xgb.train(params, d_train, 400, watchlist, early_stopping_rounds=50, verbose_eval=10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "11616998-a057-45e0-ac3c-d129e5cfec36"
      },
      "outputs": [],
      "source": [
        "d_test = xgb.DMatrix(x_test)\n",
        "p_test = bst.predict(d_test)\n",
        "\n",
        "sub = pd.DataFrame()\n",
        "sub['test_id'] = df_test['test_id']\n",
        "sub['is_duplicate'] = p_test\n",
        "sub.to_csv('simple_xgb.csv', index=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "4b74fd4a-5e89-27e1-5ddd-f170bc370697"
      },
      "source": [
        "**0.35460** on the leaderboard - a good first score!"
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}