nlp-with-disaster-tweets-bert.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.10","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Natural Language Processing with Disaster Tweets","metadata":{"id":"6H0ziAsdQfah"}},{"cell_type":"code","source":"import re                                                 # regular Expression\nimport string                                             # used for its string properties\nimport numpy as np                                        # Numpy for Linear Algebra\nimport pandas as pd                                       # Pandas for DataFrames\nfrom matplotlib import pyplot as plt                      # Matplotlib for plots\nimport seaborn as sns\n%matplotlib inline\nplt.style.use('ggplot')\n\nfrom wordcloud import WordCloud                           # Wordclouds\n\n# Sckit-Learn\nfrom sklearn.model_selection import train_test_split\n\n# Keras\nfrom keras.models import Model\nfrom keras.optimizers import Adam\nfrom keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, EarlyStopping\nfrom keras.layers import Dense, Input, Dropout\n\n# Bert Transformers\nfrom transformers import TFBertModel, BertModel\nfrom transformers import BertTokenizer\n\n# spaCy\nimport spacy\nnlp = spacy.load('en_core_web_sm')  # other options: en_core_web_md, en_core_web_lg","metadata":{"id":"L8kjhfcUQfa5","executionInfo":{"status":"ok","timestamp":1631678655239,"user_tz":-270,"elapsed":983,"user":{"displayName":"Keivan Ipchi","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiYWJZ-pareZcENvSntLEr0hS7wN6xv1TYBeXeu=s64","userId":"05399847325073645104"}},"execution":{"iopub.status.busy":"2021-11-02T08:03:55.464163Z","iopub.execute_input":"2021-11-02T08:03:55.464718Z","iopub.status.idle":"2021-11-02T08:04:05.916367Z","shell.execute_reply.started":"2021-11-02T08:03:55.464598Z","shell.execute_reply":"2021-11-02T08:04:05.915225Z"},"trusted":true},"execution_count":1,"outputs":[]},{"cell_type":"markdown","source":"## Load Data\n\n- There are 4 shared features: *id*, *keyword*, *location*, *text*\n- There is an predictive feature: *target*\n- There are 7613 entries for *training data* and 3263 for *test data*.","metadata":{"id":"w5Ma28OXQfa5"}},{"cell_type":"code","source":"# load train and test data\ntrain_df = pd.read_csv('../input/nlp-getting-started/train.csv')\ntest_df = pd.read_csv('../input/nlp-getting-started/test.csv')\n\n# shuffle and reindex\nfor df in [train_df, test_df]:    \n    df = df.sample(frac = 1).reset_index(drop = True)\n\n# print basic properties\nprint(f\"Training Data has {train_df.shape[0]} rows and {train_df.shape[1]} columns: ({list(train_df.columns)})\")\nprint(f\"Testing Data has {test_df.shape[0]} rows and {test_df.shape[1]} columns: ({list(test_df.columns)})\")","metadata":{"id":"pydPF-5xQfa5","executionInfo":{"status":"ok","timestamp":1631678657267,"user_tz":-270,"elapsed":2031,"user":{"displayName":"Keivan Ipchi","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiYWJZ-pareZcENvSntLEr0hS7wN6xv1TYBeXeu=s64","userId":"05399847325073645104"}},"execution":{"iopub.status.busy":"2021-11-02T08:04:05.918041Z","iopub.execute_input":"2021-11-02T08:04:05.918438Z","iopub.status.idle":"2021-11-02T08:04:05.998077Z","shell.execute_reply.started":"2021-11-02T08:04:05.918381Z","shell.execute_reply":"2021-11-02T08:04:05.996595Z"},"trusted":true},"execution_count":2,"outputs":[]},{"cell_type":"markdown","source":"## Exploratory Data Analysis (EDA)","metadata":{"id":"qiOFwQQHW6tF"}},{"cell_type":"markdown","source":"### Target feature\n\n- This feature breaks down into two **unbalanced** categories (use *accuracy* metric with caution):\n    - 1 (4358 records, 56%)\n    - 0 (3255 records, 44%)\n- It is suggested from [here](https://www.kaggle.com/wrrosa/keras-bert-using-tfhub-modified-train-data) that some targets have not been correctly categorized and must be fixed.","metadata":{"id":"6cwzju9hQfa6"}},{"cell_type":"code","source":"# fix mis-classified targets\nerror_ids = [328, 443, 513, 2619, 3640, 3900, 4342, 5781, 6552, 6554, 6570, 6701, 6702, 6729, 6861, 7226]\ntrain_df.loc[error_ids, 'target'] = 0\n\n# plot categories frequencies\ntrain_df['target'].plot(kind = 'hist', title = 'Target categories frequencies')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-02T08:04:06.000765Z","iopub.execute_input":"2021-11-02T08:04:06.001701Z","iopub.status.idle":"2021-11-02T08:04:06.224311Z","shell.execute_reply.started":"2021-11-02T08:04:06.001641Z","shell.execute_reply":"2021-11-02T08:04:06.223037Z"},"trusted":true},"execution_count":3,"outputs":[]},{"cell_type":"markdown","source":"### Keyword feature\nKeywords, where does this come from? How is it constructed? We don't know the answer to these two vital questions, but it seems like a good feature to work on.\n- There are 61 (<1%) missing entries on this column.\n- Spaces have been replaced by *%20* and must be fixed.\n- There are only 221 unique values out of 7491 the entries (about ~3%).\n- Duplicate entries with different formats (e.g. wrecked, wrekage and wreck) that must be fixed.\n- Some keywords (like *wound*, *volcano*, *violent storm* and ...) are all related to disaster tweets, this feature can prove useful in our final prediction/validation.","metadata":{"id":"Oje0q_hjQfa9"}},{"cell_type":"code","source":"# count null entries\nnull_entries = train_df.loc[train_df['keyword'].isnull()]\nnull_entries_count = len(null_entries)\nprint(f'Number of missing entries: {null_entries_count} (<{round(null_entries_count / train_df.shape[0] * 100)}%)')\n\n# count unique entries\nunique_locations = train_df['keyword'].unique()\nunique_locations_count = len(unique_locations)\nprint(f'Number of unique keywords: {unique_locations_count} (~{round(unique_locations_count/train_df.shape[0]*100)}%)')","metadata":{"id":"P73qT4YDQfa-","executionInfo":{"status":"ok","timestamp":1631678674290,"user_tz":-270,"elapsed":6,"user":{"displayName":"Keivan Ipchi","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiYWJZ-pareZcENvSntLEr0hS7wN6xv1TYBeXeu=s64","userId":"05399847325073645104"}},"outputId":"0d0f85b2-17cc-4b73-fce6-dfef44af85f2","execution":{"iopub.status.busy":"2021-11-02T08:04:06.226128Z","iopub.execute_input":"2021-11-02T08:04:06.226529Z","iopub.status.idle":"2021-11-02T08:04:06.236702Z","shell.execute_reply.started":"2021-11-02T08:04:06.226491Z","shell.execute_reply":"2021-11-02T08:04:06.235288Z"},"trusted":true},"execution_count":4,"outputs":[]},{"cell_type":"code","source":"# Two preprocessing steps must be done on the keyword column before further investigations:\n#    3. Manually fix a few column entries: wildfire\n#    1. Lemmatize to remove uniform all keywords\n#    2. Fix spacing formats\n\n# lemmatize text\ndef lemmatize_text(text):\n    return ' '.join([token.lemma_ for token in nlp(text)])\n\nfor df in [train_df, test_df]:\n    keyword_not_na = df['keyword'].notnull()\n    \n    # Fix column entries\n    df.loc[df['keyword'] == 'wildfire', 'keyword'] = 'wild fire'\n    \n    # Fix space encoding    \n    df.loc[keyword_not_na, 'keyword'] = df.loc[keyword_not_na, 'keyword'].apply(lambda x: x.replace('%20', ' '))\n\n    # Lemmatize\n    df.loc[keyword_not_na, 'keyword'] = df.loc[keyword_not_na, 'keyword'].apply(lemmatize_text)","metadata":{"id":"yHHanQ3GQfa-","executionInfo":{"status":"ok","timestamp":1631678756906,"user_tz":-270,"elapsed":82621,"user":{"displayName":"Keivan Ipchi","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiYWJZ-pareZcENvSntLEr0hS7wN6xv1TYBeXeu=s64","userId":"05399847325073645104"}},"execution":{"iopub.status.busy":"2021-11-02T08:04:06.238380Z","iopub.execute_input":"2021-11-02T08:04:06.238828Z","iopub.status.idle":"2021-11-02T08:05:20.831845Z","shell.execute_reply.started":"2021-11-02T08:04:06.238789Z","shell.execute_reply":"2021-11-02T08:05:20.831041Z"},"trusted":true},"execution_count":5,"outputs":[]},{"cell_type":"code","source":"# create and count aggregation on keywords and targets, sort and select top 25 keywords\nn_keywords = 25\n\nkt_agg = train_df[['keyword', 'target', 'text']].groupby(\n    by = ['keyword', 'target'],\n    as_index = False\n).count().sort_values(\n    by = ['keyword', 'target', 'text'],\n    ascending = False\n).reset_index(drop = True).iloc[:n_keywords * 2]\n\n# prepare data to be plot on stacked-bar plot\nkt_agg_X, kt_agg_Y0, kt_agg_Y1 = [], [], []\nfor i in range(kt_agg.shape[0] - 1):\n    \n    row_1 = kt_agg.iloc[i]\n    row_0 = kt_agg.iloc[i + 1]\n    \n    kt_agg_X.append(row_0['keyword'])\n    \n    if row_0['keyword'] == row_1['keyword']:\n        kt_agg_Y0.append(row_0['text'])\n        kt_agg_Y1.append(row_1['text'])\n    elif row_0['target'] == 1:\n        kt_agg_Y0.append(row_0['text'])\n        kt_agg_Y1.append(0)\n    else:\n        kt_agg_Y0.append(0)\n        kt_agg_Y1.append(row_1['text'])\n\n# plot the stack-bar\nfig = plt.figure(figsize = (20, 5))\nplt.bar(x = kt_agg_X, height = kt_agg_Y0, color = 'b', label = 'Non-Disaster')\nplt.bar(x = kt_agg_X, height = kt_agg_Y1, color = 'g', label = 'Disaster')\nplt.title(\"Keywords/Disastrous state count\")\nplt.xticks(rotation = 20)\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-02T08:05:20.833157Z","iopub.execute_input":"2021-11-02T08:05:20.834095Z","iopub.status.idle":"2021-11-02T08:05:21.336255Z","shell.execute_reply.started":"2021-11-02T08:05:20.834032Z","shell.execute_reply":"2021-11-02T08:05:21.335021Z"},"trusted":true},"execution_count":6,"outputs":[]},{"cell_type":"markdown","source":"### Location feature\n\nI honestly don't like this feature, because I don't know where it comes from and how it is constructed. Is it the location of the user making the tweet? If so, then it won't be of any help. (Imagine you are in USA and tweeting about a disaster in Germany!)\n\nLmao; it also has all kinds of weird entries in it which makes matters worse, for instance: *Live On Webcam*, *milky way*, *Twitter Lockout in progress*, *Your Sister's Bedroom*.","metadata":{}},{"cell_type":"code","source":"for df in [train_df, test_df]:\n    df.drop(columns = ['location'], inplace = True)","metadata":{"execution":{"iopub.status.busy":"2021-11-02T08:05:21.338120Z","iopub.execute_input":"2021-11-02T08:05:21.338944Z","iopub.status.idle":"2021-11-02T08:05:21.348321Z","shell.execute_reply.started":"2021-11-02T08:05:21.338851Z","shell.execute_reply":"2021-11-02T08:05:21.346800Z"},"trusted":true},"execution_count":7,"outputs":[]},{"cell_type":"markdown","source":"### Text feature\nThis is the most important feature of the dataset. Having a quick look at it, I figured:\n- Extracting **hashtags** sounds like a good idea.\n- **mentions** must be removed as usernames aren't necessary meaningful.\n- **Links** can be scraped. This can be a potentially a good source of extra information.\n- Removing **Emojies** and **non-ASCII characters** are helpful.\n- Words with too few characters must be removed (careful with this one)\n- **HTML Tags** must be removed.\n- Fix **abbreviations** in the text. (dictionary of abbreviations taken from [here](https://www.kaggle.com/rftexas/text-only-kfold-bert))\n\nAlternative preprocessings:\n- **Lemmatize** all words (reduces dimensionality of the data)\n- Remove all **stopwords** (also reduces dimensionality)\n- Remove **punctuation** (depends on the model you are using)\n\nNOTE: What preprocessings you apply on you model purely depend on the model you are developing. Whether Dense, LSTM or Embedding Layers are used preprocessing methods should also change accordingly.","metadata":{"id":"Z4byYE9CbclS"}},{"cell_type":"code","source":"# Extract hashtags in form of #<str> (They must have atleast 3 characters)\ndef extract_hashtags(text):\n    matches = re.findall(r'#\\w*', text)\n    return [match[1:].lower() for match in matches if len(match) > 2] if matches != [] else np.nan\n\n# Extract links\ndef extract_links(text):\n    matches = re.findall(r'\\bhttps?://\\S+', text)\n    return matches if matches != [] else np.nan\n\nfor df in [train_df, test_df]:\n    # Feature extract #hashtags, @mentions and :links\n    df['hashtags'] = df['text'].apply(extract_hashtags)\n    df['links'] = df['text'].apply(extract_links)    ","metadata":{"id":"Kn0EzUGiQfa_","executionInfo":{"status":"ok","timestamp":1631678763589,"user_tz":-270,"elapsed":323,"user":{"displayName":"Keivan Ipchi","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiYWJZ-pareZcENvSntLEr0hS7wN6xv1TYBeXeu=s64","userId":"05399847325073645104"}},"execution":{"iopub.status.busy":"2021-11-02T08:05:21.352200Z","iopub.execute_input":"2021-11-02T08:05:21.352772Z","iopub.status.idle":"2021-11-02T08:05:21.631914Z","shell.execute_reply.started":"2021-11-02T08:05:21.352709Z","shell.execute_reply":"2021-11-02T08:05:21.630858Z"},"trusted":true},"execution_count":8,"outputs":[]},{"cell_type":"code","source":"abbreviations = {\n    \"$\" : \" dollar \",\n    \"€\" : \" euro \",\n    \"4ao\" : \"for adults only\",\n    \"a.m\" : \"before midday\",\n    \"a3\" : \"anytime anywhere anyplace\",\n    \"aamof\" : \"as a matter of fact\",\n    \"acct\" : \"account\",\n    \"adih\" : \"another day in hell\",\n    \"afaic\" : \"as far as i am concerned\",\n    \"afaict\" : \"as far as i can tell\",\n    \"afaik\" : \"as far as i know\",\n    \"afair\" : \"as far as i remember\",\n    \"afk\" : \"away from keyboard\",\n    \"app\" : \"application\",\n    \"approx\" : \"approximately\",\n    \"apps\" : \"applications\",\n    \"asap\" : \"as soon as possible\",\n    \"asl\" : \"age, sex, location\",\n    \"atk\" : \"at the keyboard\",\n    \"ave.\" : \"avenue\",\n    \"aymm\" : \"are you my mother\",\n    \"ayor\" : \"at your own risk\", \n    \"b&b\" : \"bed and breakfast\",\n    \"b+b\" : \"bed and breakfast\",\n    \"b.c\" : \"before christ\",\n    \"b2b\" : \"business to business\",\n    \"b2c\" : \"business to customer\",\n    \"b4\" : \"before\",\n    \"b4n\" : \"bye for now\",\n    \"b@u\" : \"back at you\",\n    \"bae\" : \"before anyone else\",\n    \"bak\" : \"back at keyboard\",\n    \"bbbg\" : \"bye bye be good\",\n    \"bbc\" : \"british broadcasting corporation\",\n    \"bbias\" : \"be back in a second\",\n    \"bbl\" : \"be back later\",\n    \"bbs\" : \"be back soon\",\n    \"be4\" : \"before\",\n    \"bfn\" : \"bye for now\",\n    \"blvd\" : \"boulevard\",\n    \"bout\" : \"about\",\n    \"brb\" : \"be right back\",\n    \"bros\" : \"brothers\",\n    \"brt\" : \"be right there\",\n    \"bsaaw\" : \"big smile and a wink\",\n    \"btw\" : \"by the way\",\n    \"bwl\" : \"bursting with laughter\",\n    \"c/o\" : \"care of\",\n    \"cet\" : \"central european time\",\n    \"cf\" : \"compare\",\n    \"cia\" : \"central intelligence agency\",\n    \"csl\" : \"can not stop laughing\",\n    \"cu\" : \"see you\",\n    \"cul8r\" : \"see you later\",\n    \"cv\" : \"curriculum vitae\",\n    \"cwot\" : \"complete waste of time\",\n    \"cya\" : \"see you\",\n    \"cyt\" : \"see you tomorrow\",\n    \"dae\" : \"does anyone else\",\n    \"dbmib\" : \"do not bother me i am busy\",\n    \"diy\" : \"do it yourself\",\n    \"dm\" : \"direct message\",\n    \"dwh\" : \"during work hours\",\n    \"e123\" : \"easy as one two three\",\n    \"eet\" : \"eastern european time\",\n    \"eg\" : \"example\",\n    \"embm\" : \"early morning business meeting\",\n    \"encl\" : \"enclosed\",\n    \"encl.\" : \"enclosed\",\n    \"etc\" : \"and so on\",\n    \"faq\" : \"frequently asked questions\",\n    \"fawc\" : \"for anyone who cares\",\n    \"fb\" : \"facebook\",\n    \"fc\" : \"fingers crossed\",\n    \"fig\" : \"figure\",\n    \"fimh\" : \"forever in my heart\", \n    \"ft.\" : \"feet\",\n    \"ft\" : \"featuring\",\n    \"ftl\" : \"for the loss\",\n    \"ftw\" : \"for the win\",\n    \"fwiw\" : \"for what it is worth\",\n    \"fyi\" : \"for your information\",\n    \"g9\" : \"genius\",\n    \"gahoy\" : \"get a hold of yourself\",\n    \"gal\" : \"get a life\",\n    \"gcse\" : \"general certificate of secondary education\",\n    \"gfn\" : \"gone for now\",\n    \"gg\" : \"good game\",\n    \"gl\" : \"good luck\",\n    \"glhf\" : \"good luck have fun\",\n    \"gmt\" : \"greenwich mean time\",\n    \"gmta\" : \"great minds think alike\",\n    \"gn\" : \"good night\",\n    \"g.o.a.t\" : \"greatest of all time\",\n    \"goat\" : \"greatest of all time\",\n    \"goi\" : \"get over it\",\n    \"gps\" : \"global positioning system\",\n    \"gr8\" : \"great\",\n    \"gratz\" : \"congratulations\",\n    \"gyal\" : \"girl\",\n    \"h&c\" : \"hot and cold\",\n    \"hp\" : \"horsepower\",\n    'hwy': 'highway',\n    \"hr\" : \"hour\",\n    \"hrh\" : \"his royal highness\",\n    \"ht\" : \"height\",\n    \"ibrb\" : \"i will be right back\",\n    \"ic\" : \"i see\",\n    \"icq\" : \"i seek you\",\n    \"icymi\" : \"in case you missed it\",\n    \"idc\" : \"i do not care\",\n    \"idgadf\" : \"i do not give a damn fuck\",\n    \"idgaf\" : \"i do not give a fuck\",\n    \"idk\" : \"i do not know\",\n    \"ie\" : \"that is\",\n    \"i.e\" : \"that is\",\n    \"ifyp\" : \"i feel your pain\",\n    \"IG\" : \"instagram\",\n    \"iirc\" : \"if i remember correctly\",\n    \"ilu\" : \"i love you\",\n    \"ily\" : \"i love you\",\n    \"imho\" : \"in my humble opinion\",\n    \"imo\" : \"in my opinion\",\n    \"imu\" : \"i miss you\",\n    \"iow\" : \"in other words\",\n    \"irl\" : \"in real life\",\n    \"j4f\" : \"just for fun\",\n    \"jic\" : \"just in case\",\n    \"jk\" : \"just kidding\",\n    \"jsyk\" : \"just so you know\",\n    \"l8r\" : \"later\",\n    \"lb\" : \"pound\",\n    \"lbs\" : \"pounds\",\n    \"ldr\" : \"long distance relationship\",\n    \"lmao\" : \"laugh my ass off\",\n    \"lmfao\" : \"laugh my fucking ass off\",\n    \"lol\" : \"laughing out loud\",\n    \"ltd\" : \"limited\",\n    \"ltns\" : \"long time no see\",\n    \"m8\" : \"mate\",\n    \"mf\" : \"motherfucker\",\n    \"mfs\" : \"motherfuckers\",\n    \"mfw\" : \"my face when\",\n    \"mofo\" : \"motherfucker\",\n    \"mph\" : \"miles per hour\",\n    \"mr\" : \"mister\",\n    \"mrw\" : \"my reaction when\",\n    \"ms\" : \"miss\",\n    \"mte\" : \"my thoughts exactly\",\n    \"nagi\" : \"not a good idea\",\n    \"nbc\" : \"national broadcasting company\",\n    \"nbd\" : \"not big deal\",\n    \"nfs\" : \"not for sale\",\n    \"ngl\" : \"not going to lie\",\n    \"nhs\" : \"national health service\",\n    \"nrn\" : \"no reply necessary\",\n    \"nsfl\" : \"not safe for life\",\n    \"nsfw\" : \"not safe for work\",\n    \"nth\" : \"nice to have\",\n    \"nvr\" : \"never\",\n    \"nyc\" : \"new york city\",\n    \"oc\" : \"original content\",\n    \"og\" : \"original\",\n    \"ohp\" : \"overhead projector\",\n    \"oic\" : \"oh i see\",\n    \"omdb\" : \"over my dead body\",\n    \"omg\" : \"oh my god\",\n    \"omw\" : \"on my way\",\n    \"p.a\" : \"per annum\",\n    \"p.m\" : \"after midday\",\n    \"pm\" : \"prime minister\",\n    \"poc\" : \"people of color\",\n    \"pov\" : \"point of view\",\n    \"pp\" : \"pages\",\n    \"ppl\" : \"people\",\n    \"prw\" : \"parents are watching\",\n    \"ps\" : \"postscript\",\n    \"pt\" : \"point\",\n    \"ptb\" : \"please text back\",\n    \"pto\" : \"please turn over\",\n    \"qpsa\" : \"what happens\",\n    \"ratchet\" : \"rude\",\n    \"rbtl\" : \"read between the lines\",\n    \"rlrt\" : \"real life retweet\", \n    \"rofl\" : \"rolling on the floor laughing\",\n    \"roflol\" : \"rolling on the floor laughing out loud\",\n    \"rotflmao\" : \"rolling on the floor laughing my ass off\",\n    \"rt\" : \"retweet\",\n    \"ruok\" : \"are you ok\",\n    \"sfw\" : \"safe for work\",\n    \"sk8\" : \"skate\",\n    \"smh\" : \"shake my head\",\n    \"sq\" : \"square\",\n    \"srsly\" : \"seriously\", \n    \"ssdd\" : \"same stuff different day\",\n    \"tbh\" : \"to be honest\",\n    \"tbs\" : \"tablespooful\",\n    \"tbsp\" : \"tablespooful\",\n    \"tfw\" : \"that feeling when\",\n    \"thks\" : \"thank you\",\n    \"tho\" : \"though\",\n    \"thx\" : \"thank you\",\n    \"tia\" : \"thanks in advance\",\n    \"til\" : \"today i learned\",\n    \"tl;dr\" : \"too long i did not read\",\n    \"tldr\" : \"too long i did not read\",\n    \"tmb\" : \"tweet me back\",\n    \"tntl\" : \"trying not to laugh\",\n    \"ttyl\" : \"talk to you later\",\n    \"u\" : \"you\",\n    \"u2\" : \"you too\",\n    \"u4e\" : \"yours for ever\",\n    \"utc\" : \"coordinated universal time\",\n    \"w/\" : \"with\",\n    \"w/o\" : \"without\",\n    'w/e': 'whatever',\n    \"w8\" : \"wait\",\n    \"wassup\" : \"what is up\",\n    \"wb\" : \"welcome back\",\n    \"wtf\" : \"what the fuck\",\n    \"wtg\" : \"way to go\",\n    \"wtpa\" : \"where the party at\",\n    \"wuf\" : \"where are you from\",\n    \"wuzup\" : \"what is up\",\n    \"wywh\" : \"wish you were here\",\n    \"yd\" : \"yard\",\n    \"ygtr\" : \"you got that right\",\n    \"ynk\" : \"you never know\",\n    \"zzz\" : \"sleeping bored and tired\"\n}\n\n# Change an abbreviation to its true word\ndef fix_abbrev(text):\n    return ' '.join([abbreviations[word.lower()] if (word.lower() in abbreviations.keys()) else word for word in text.split()])\n\n# Replace some others smiley face with SADFACE\ndef transcription_sad(text):\n    smiley = re.compile(r'[8:=;][\\'\\-]?[(\\\\/]')\n    return smiley.sub(r'sad', text)\n\n# Replace <3 with HEART\ndef transcription_heart(text):\n    heart = re.compile(r'<3')\n    return heart.sub(r'love', text)\n\n# Replace URLs\ndef remove_urls(text):\n    return re.sub(r'https?://\\S+|www\\.\\S+', '', text)\n\n# Remove HTML\ndef remove_html(text):\n    return re.sub(r'<.*?>', '', text)\n\n# Converts text to lowercase\ndef to_lower(text):\n    return text.lower()\n\n# Remove words contaning numbers\ndef remove_numbers(text):\n    return re.sub(r'\\w*\\d\\w*', '', text)\n\n# Remove text in brackets\ndef remove_brackets(text):\n    return re.sub(r'\\[.*?\\]', '', text)  \n\n# Replace mentions\ndef remove_mentions(text):\n    return re.sub(r'@\\w*', '', text)\n\n# Remove hashtags\ndef remove_hashtags(text):\n    return re.sub(r'#\\w*', '', text)\n\n# Remove emojis\ndef remove_emojis(text):\n    regrex_pattern = re.compile(pattern = \"[\"\n        u\"\\U0001F600-\\U0001F64F\"\n        u\"\\U0001F300-\\U0001F5FF\"\n        u\"\\U0001F680-\\U0001F6FF\"\n        u\"\\U0001F1E0-\\U0001F1FF\"\n    \"]+\", flags = re.UNICODE)\n    return regrex_pattern.sub(r'',text)\n\n# Remove non-ASCII characters\ndef remove_non_ascii(text):\n#     return ''.join(filter(lambda x: x in string.printable, text))\n    return text.encode(\"ascii\",errors=\"ignore\").decode()\n\n# Remove stopwords\ndef remove_stopwords(text):\n    return ' '.join([token.text for token in nlp(text) if not token.is_stop])\n\n# Remove punctuation\ndef remove_punctuation(text):\n    table = str.maketrans('', '', string.punctuation)\n    return text.translate(table)\n\n# Remove white space (Extra step, just in case)\ndef remove_whitespace(text):\n    return ' '.join(text.split())\n\n# Filter out words with too few characters (2 by default)\ndef filter_words(text):\n    return ' '.join([word for word in text.split() if len(word) > 2])","metadata":{"id":"aXTrxX1lXr4i","executionInfo":{"status":"ok","timestamp":1631678771227,"user_tz":-270,"elapsed":753,"user":{"displayName":"Keivan Ipchi","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiYWJZ-pareZcENvSntLEr0hS7wN6xv1TYBeXeu=s64","userId":"05399847325073645104"}},"execution":{"iopub.status.busy":"2021-11-02T08:05:21.633841Z","iopub.execute_input":"2021-11-02T08:05:21.634208Z","iopub.status.idle":"2021-11-02T08:05:21.664348Z","shell.execute_reply.started":"2021-11-02T08:05:21.634175Z","shell.execute_reply":"2021-11-02T08:05:21.663480Z"},"trusted":true},"execution_count":9,"outputs":[]},{"cell_type":"code","source":"for df in [train_df, test_df]:\n    \n    text = df['text']\n    \n    # Convert to lowercase\n    text = text.apply(to_lower)\n\n    # Replace symbols\n    text = text.replace(r'&amp;?', r'and')\n    text = text.replace(r'&lt;', r'<')\n    text = text.replace(r'&gt;', r'>')\n    text = text.replace('&amp;', \" and \")\n    \n    # Manual Lemmatize\n    text = text.str.replace('won\\'t', 'will not')\n    text = text.str.replace('can\\'t', 'cannot')\n    text = text.str.replace('i\\'m', 'i am')\n    text = text.replace('ain\\'t', 'is not')\n    \n    # Remove mentions and links (hashtags too?)\n#     text = text.apply(remove_hashtags)\n    text = text.apply(remove_mentions)\n    text = text.apply(remove_urls)\n\n    # Fix abbreviations\n    text = text.apply(fix_abbrev)\n    \n    # Remove HTML tags\n    text = text.apply(remove_html)\n    \n    # Remove texts within brackets\n#     text = text.apply(remove_brackets)\n\n    # Fix emojies\n    text = text.apply(transcription_sad)   # Sad emojies\n    text = text.apply(transcription_heart) # Heart emoji\n    text = text.apply(remove_emojis)       # General emojies\n\n    # Remove non-ASCII characters\n    text = text.apply(remove_non_ascii)\n    \n    # Remove words contaning numbers\n    text = text.apply(remove_numbers)\n    \n    # Remove stopwords\n#     text = text.apply(remove_stopwords)\n    \n    # Remove punctuations\n    text = text.apply(remove_punctuation)\n\n    # Lemmatize text\n#     text = text.apply(lemmatize_text)\n\n    # Remove words with few characters    \n#     text = text.apply(filter_words)\n    \n    # Fill entry if turns out empty\n    text = text.apply(lambda x: x if x != '' else '?')\n\n    df['clean_text'] = text","metadata":{"id":"3DhdYv0GQfbA","executionInfo":{"status":"ok","timestamp":1631678983117,"user_tz":-270,"elapsed":211891,"user":{"displayName":"Keivan Ipchi","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiYWJZ-pareZcENvSntLEr0hS7wN6xv1TYBeXeu=s64","userId":"05399847325073645104"}},"outputId":"66b6dfbc-e6fb-48df-c7ee-123d2d029f54","execution":{"iopub.status.busy":"2021-11-02T08:05:21.665702Z","iopub.execute_input":"2021-11-02T08:05:21.666206Z","iopub.status.idle":"2021-11-02T08:05:22.050525Z","shell.execute_reply.started":"2021-11-02T08:05:21.666161Z","shell.execute_reply":"2021-11-02T08:05:22.049572Z"},"trusted":true},"execution_count":10,"outputs":[]},{"cell_type":"code","source":"fig, axs = plt.subplots(nrows = 1, ncols = 2, figsize = (20, 7))\n\n# Disaster Tweets\nword_cloud = WordCloud()\nword_cloud.generate(\" \".join(train_df.loc[train_df['target'] == 1, 'clean_text']))\naxs[0].imshow(word_cloud)\naxs[0].set_title(\"Disaster Tweets (Clean text)\")\naxs[0].axis('off')\n\n# Normal Tweets\nword_cloud = WordCloud()\nword_cloud.generate(\" \".join(train_df.loc[train_df['target'] == 0, 'clean_text']))\naxs[1].set_title(\"Non-Disaster Tweets (Clean text)\")\naxs[1].imshow(word_cloud)\naxs[1].axis('off')\n\nplt.show()","metadata":{"id":"Hw8vMHgAQfbA","executionInfo":{"status":"ok","timestamp":1631678768689,"user_tz":-270,"elapsed":2666,"user":{"displayName":"Keivan Ipchi","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiYWJZ-pareZcENvSntLEr0hS7wN6xv1TYBeXeu=s64","userId":"05399847325073645104"}},"outputId":"a7bea7e6-8f7e-4e05-ee0f-aea0b16a96bd","execution":{"iopub.status.busy":"2021-11-02T08:05:22.051759Z","iopub.execute_input":"2021-11-02T08:05:22.052327Z","iopub.status.idle":"2021-11-02T08:05:23.270151Z","shell.execute_reply.started":"2021-11-02T08:05:22.052275Z","shell.execute_reply":"2021-11-02T08:05:23.269245Z"},"trusted":true},"execution_count":11,"outputs":[]},{"cell_type":"markdown","source":"## Develop the Model","metadata":{"id":"n97hmHXxQfbB"}},{"cell_type":"code","source":"# encode data for BERT and create attention masks\n\nTOKENIZER = BertTokenizer.from_pretrained(\"bert-base-uncased\")\n\ndef get_BERT_encode(data, maximum_len) :\n    input_ids = []\n    attention_masks = []\n  \n    for i in range(len(data['clean_text'])):\n        encoded = TOKENIZER.encode_plus(data['clean_text'][i],\n                                        add_special_tokens = True,\n                                        max_length = maximum_len,\n                                        pad_to_max_length = True,\n                                        return_attention_mask = True)\n      \n        input_ids.append(encoded['input_ids'])\n        attention_masks.append(encoded['attention_mask'])\n        \n    return np.array(input_ids), np.array(attention_masks)\n\ntrain_input_ids, train_attention_masks = get_BERT_encode(train_df, 60)\ntest_input_ids, test_attention_masks = get_BERT_encode(test_df, 60)\n\ntrain_X = [train_input_ids, train_attention_masks]\ntest_X = [test_input_ids, test_attention_masks]","metadata":{"execution":{"iopub.status.busy":"2021-11-02T08:14:22.600239Z","iopub.execute_input":"2021-11-02T08:14:22.601797Z","iopub.status.idle":"2021-11-02T08:14:32.771050Z","shell.execute_reply.started":"2021-11-02T08:14:22.601713Z","shell.execute_reply":"2021-11-02T08:14:32.770021Z"},"trusted":true},"execution_count":13,"outputs":[]},{"cell_type":"code","source":"# define model architecture\n\nbert_base = TFBertModel.from_pretrained('bert-base-uncased')\n\ndef get_BERT_model(model_layer, learning_rate):\n    \n    input_ids = Input(shape = (60,), dtype = 'int32')\n    attention_masks = Input(shape = (60,), dtype = 'int32')    \n\n    output = model_layer([input_ids, attention_masks])[1]\n#     output = tf.keras.layers.Dense(64, activation = 'relu')(output)\n    output = Dropout(0.2)(output)    \n    output = Dense(units = 1, activation = 'sigmoid')(output)\n    \n    model = Model(inputs = [input_ids,attention_masks],outputs = output)\n    model.compile(Adam(lr = learning_rate), loss = 'binary_crossentropy', metrics = ['accuracy'])\n    \n    return model\n\nmodel = get_BERT_model(bert_base, learning_rate = 1e-5)\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2021-11-02T08:14:32.773073Z","iopub.execute_input":"2021-11-02T08:14:32.773494Z","iopub.status.idle":"2021-11-02T08:14:56.357144Z","shell.execute_reply.started":"2021-11-02T08:14:32.773454Z","shell.execute_reply":"2021-11-02T08:14:56.355867Z"},"trusted":true},"execution_count":14,"outputs":[]},{"cell_type":"markdown","source":"### Training & Evaluation","metadata":{}},{"cell_type":"code","source":"# define training parameters\nepochs = 5\nbatch_size = 16\nvalidation_split = 0.1\n\n# define training callbacks\ncheckpoint = ModelCheckpoint(\n    filepath = 'model.h5',\n    verbose = 1,\n    save_best_only = True,\n    save_weights_only = True\n)\nreduce_lr = ReduceLROnPlateau(\n    factor = 0.2, \n    verbose = 1, \n    patience = 5,                        \n    min_lr = 0.001\n)\nearly_stopping = EarlyStopping(patience = 3)\n\n# train the model\nhistory = model.fit(\n    x = train_X,\n    y = train_df['target'],\n    epochs = epochs,\n    batch_size = batch_size,\n    validation_split = validation_split,\n    callbacks = [reduce_lr, checkpoint, early_stopping]  # All callbacks monitor val_loss by default\n)","metadata":{"id":"ziZ6ZkiHQfbC","executionInfo":{"status":"ok","timestamp":1631679573333,"user_tz":-270,"elapsed":6558,"user":{"displayName":"Keivan Ipchi","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiYWJZ-pareZcENvSntLEr0hS7wN6xv1TYBeXeu=s64","userId":"05399847325073645104"}},"outputId":"10168fac-8d33-4051-805a-102bc415f0ca","execution":{"iopub.status.busy":"2021-11-02T08:16:24.500327Z","iopub.execute_input":"2021-11-02T08:16:24.501047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot all the metrics used for training the model\nfig, axs = plt.subplots(1, len(METRICS), figsize = (18, 5))\n\nfor i, metric in enumerate(METRICS):\n    axs[i].set_title('Analysis of ' + metric)\n    axs[i].plot(range(EPOCHS), history.history[metric], label = metric)\n    axs[i].plot(range(EPOCHS), history.history[f'val_' + metric], label = f'val_' + metric)\n    axs[i].legend()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Making a Prediction\nThe final model used for prediction is the one with best *val_loss* metric. (The selection is taken care of by **ModelCheckpoint**).","metadata":{"id":"spbsvsJoQfbD"}},{"cell_type":"code","source":"# load best resulting model\nmodel.load_weights('model.h5')\n\n# make predictions\npred = model.predict(test_X, verbose = 2)\npred = np.round(pred).astype(int).reshape(pred.shape[0])\n\n# create a submission\nsubmission = pd.DataFrame({'id': test_df['id'], 'target': pred})\nsubmission.to_csv(f'./submission.csv', index = False)","metadata":{"id":"v2sfkN-PQfbD","executionInfo":{"status":"ok","timestamp":1631678991216,"user_tz":-270,"elapsed":814,"user":{"displayName":"Keivan Ipchi","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiYWJZ-pareZcENvSntLEr0hS7wN6xv1TYBeXeu=s64","userId":"05399847325073645104"}},"outputId":"129805d5-83c6-4212-d508-bab3642cc8d8","trusted":true},"execution_count":null,"outputs":[]}]}