Analysis_GPT2_Classifcation_September2024_revised.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Classification and Analysis\n",
    "Now we're at the point where we should be able to:\n",
    "* Read in a collection of documents - a *corpus* which contains the manually coded and predicted papers\n",
    "* Do some preliminary data analysis and select only papers >= 2006\n",
    "* Load the Spacy language model\n",
    "* Do some text processing and lemmatize abstract using the Spcy lemmatizer\n",
    "* Use TFIDF to fit and tranform the vectorized texts\n",
    "* Implement NMF for topic modelling\n",
    "* Implement t-SNE to visualize the topics\n",
    "* Implement geoparsing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perform imports and load the dataset\n",
    "The seen dataset contains the bibliometric information of over 1600 publications which have been manually labelled by two authors. Inconsistencies have been discussed and agreed upon. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Tuple\n",
    "import copy as cp\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from wordcloud import WordCloud"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### TODO Add your file path ###\n",
    "file_path = '../../Data/ClimateEducation/'\n",
    "#File that was extracted from the big database having only labelled data. Relevant = 1 means relevant.\n",
    "file_name = 'output_relevant_unseen_seen_merge_shuffle_August2023.xlsx'\n",
    "df = pd.read_excel(file_path + file_name)\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.rename(columns={'Unnamed: 0': 'Identifier'}, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['PubYear'] = df['PubYear'].astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Take a look at a typical abstract."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detect & remove empty strings\n",
    "Technically, we're dealing with \"whitespace only\" strings. If the original .xlsx file had contained empty strings, pandas **.read_xlsx()** would have assigned NaN values to those cells by default.\n",
    "\n",
    "In order to detect these strings we need to iterate over each row in the DataFrame. The **.itertuples()** pandas method is a good tool for this as it provides access to every field. For brevity we'll assign the names `i`, `lb` and `rv` to the `index`, `DOI` and `Title` columns, but actually we need to define over which columns we want to irerate and the code below needs to reflect this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Take a quick look at the `label` column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = df['Relevant'].value_counts().plot.bar(figsize = (5,7), color=['orange', 'green'])\n",
    "ax.set_xticklabels(['Irrelevant','Relevant'], rotation = 45, fontsize=12)\n",
    "ax.set_ylabel(\"Number of papers\", fontsize=12)\n",
    "ax.set_title('Papers split',fontsize=14)\n",
    "plt.savefig('../../Data/ClimateEducation/Figures_092024/irrelevant_relevant_Split.svg', dpi='figure',format = 'svg',\n",
    "        bbox_inches='tight', pad_inches=0.1,\n",
    "        facecolor='auto', edgecolor='auto',\n",
    "        backend=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr = df[df['Relevant'] == 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr.reset_index(inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(18,6))\n",
    "subgrade_order = sorted(npr['PubYear'].dropna().astype(int).unique())\n",
    "x1 = npr['PubYear'].dropna().astype(int)\n",
    "chart = sns.countplot(x=x1,data=npr,order = subgrade_order, palette = 'coolwarm')\n",
    "chart.set_xticklabels(chart.get_xticklabels(), rotation=90)\n",
    "\n",
    "None\n",
    "#npr['PubYear'].value_counts().plot(kind = 'bar', figsize = (12,6))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib.ticker import FormatStrFormatter\n",
    "plt.figure(figsize=(18,6))\n",
    "subgrade_order = sorted(npr[npr['PubYear']>1960]['PubYear'].dropna().astype(int).unique())\n",
    "x1 = npr['PubYear'].dropna().astype(int)\n",
    "chart = sns.countplot(x=x1,data=npr,order = subgrade_order, palette = 'coolwarm')\n",
    "chart.set_xticklabels(chart.get_xticklabels(), rotation=90, size =15)\n",
    "chart.set_yticklabels(chart.get_yticks(), size = 15, )\n",
    "chart.set_title(\"Publications per year\", fontsize=20)\n",
    "chart.yaxis.set_major_formatter(FormatStrFormatter('%.0f'))\n",
    "chart.set_xlabel(xlabel = None)\n",
    "chart.set_ylabel(ylabel = None)\n",
    "plt.savefig('../../Data/ClimateEducation/Figures_092024/PubYear_All.eps', dpi='figure',format = 'eps',\n",
    "        bbox_inches='tight', pad_inches=0.1,\n",
    "        facecolor='auto', edgecolor='auto',\n",
    "        backend=None)\n",
    "None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Given the low number of papers before 2008 we keep only from 2008 onwards\n",
    "npr_2008 = npr[npr['PubYear'] >=2008]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(npr[npr['PubYear'] >= 2008])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m spacy validate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Importa libraries for semantic analysis and other tasks\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "import spacy\n",
    "nlp = spacy.load(\"en_core_web_lg\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp.Defaults.stop_words |= {\"climate\",\"change\",\"education\", \"climatic\",\"changes\",\"climat\", \"changing\", \"chang\",\n",
    "        \"educ\", \"educational\", \"educative\", \"teach\", \"teaching\",\"global\", \"warming\", \"die\", \n",
    "                           \"und\", \"das\", \"pro\", \"auf\",'ll','ve', 'der', 'ein','sich', 'für'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "def britishize(string):\n",
    "    url =\"https://raw.githubusercontent.com/hyperreality/American-British-English-Translator/master/data/american_spellings.json\"\n",
    "    american_to_british_dict = requests.get(url).json()    \n",
    "\n",
    "    for american_spelling, british_spelling in american_to_british_dict.items():\n",
    "        string = string.replace(american_spelling, british_spelling)\n",
    "  \n",
    "    return string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#We want all abstracts in british english\n",
    "#Need a good network to run this because of the url request in the britishize function. \n",
    "npr_2008['Abstract_british'] = npr_2008['Abstract'].apply(lambda x: britishize(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Use the spacy lemmatizer to get the best result so far\n",
    "npr_2008['abstract_lemmatized'] = npr_2008['Abstract_british'].apply(lambda row: \" \".join([w.lemma_ for w in nlp(row)]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_words_2 = nlp.Defaults.stop_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#tfidf = TfidfVectorizer(max_df=0.95, min_df=2, ngram_range = (1,1), stop_words=stop_words_2)\n",
    "#tfidf = TfidfVectorizer(max_df=0.95, min_df=2, ngram_range = (1,2), stop_words=stop_words_2)\n",
    "#tfidf = TfidfVectorizer(max_df=0.95, min_df=2, ngram_range = (1,3), stop_words=stop_words_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#if ngram_22\n",
    "tfidf = TfidfVectorizer(max_df=0.95, min_df=2, ngram_range=(2,2), stop_words=list(stop_words_2))\n",
    "dtm = tfidf.fit_transform(npr_2008['abstract_lemmatized'])\n",
    "tfidf_weights_22 = [(word, dtm.getcol(idx).sum()) for word, idx in tfidf.vocabulary_.items()]\n",
    "feature_names_22 = tfidf.get_feature_names_out()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#if ngram_12 - which is the one used throughout the topic modelling\n",
    "tfidf = TfidfVectorizer(max_df=0.95, min_df=2, ngram_range=(1,2), stop_words=list(stop_words_2))\n",
    "dtm = tfidf.fit_transform(npr_2008['abstract_lemmatized'])\n",
    "tfidf_weights_12 = [(word, dtm.getcol(idx).sum()) for word, idx in tfidf.vocabulary_.items()]\n",
    "feature_names_12 = tfidf.get_feature_names_out()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from matplotlib.ticker import FormatStrFormatter\n",
    "\n",
    "# Create a single figure for the count plot and word clouds\n",
    "fig = plt.figure(figsize=(12, 10))\n",
    "\n",
    "# Add a subplot for the count plot (spanning the full width)\n",
    "ax1 = fig.add_subplot(2, 1, 1)  # 2 rows, 1 column, first subplot\n",
    "x1 = npr_2008['PubYear'].dropna().astype(int)\n",
    "subgrade_order = sorted(npr_2008['PubYear'].dropna().astype(int).unique())\n",
    "chart = sns.countplot(x=x1, data=npr_2008, order=subgrade_order, palette='coolwarm', ax=ax1)\n",
    "chart.set_xticklabels(chart.get_xticklabels(), rotation=90, size=12)\n",
    "chart.set_yticklabels(chart.get_yticks(), size=12)\n",
    "chart.set_title(\"Publications per Year from 2008\", fontsize=16)\n",
    "chart.yaxis.set_major_formatter(FormatStrFormatter('%.0f'))\n",
    "chart.set_xlabel(xlabel=None)\n",
    "chart.set_ylabel('Count of Publications', fontsize=12)\n",
    "\n",
    "# Add subplots for the word clouds (2 side by side)\n",
    "ax2 = fig.add_subplot(2, 2, 3)  # 2 rows, 2 columns, first column of second row\n",
    "ax2.imshow(w12, interpolation='bilinear')\n",
    "#ax2.set_title('Word Cloud 1', fontsize=16)\n",
    "ax2.axis('off')  # Hide axis\n",
    "\n",
    "ax3 = fig.add_subplot(2, 2, 4)  # 2 rows, 2 columns, second column of second row\n",
    "ax3.imshow(w22, interpolation='bilinear')\n",
    "#ax3.set_title('Word Cloud 2', fontsize=16)\n",
    "ax3.axis('off')  # Hide axis\n",
    "\n",
    "# Adjust layout to look better\n",
    "plt.tight_layout(pad=2)\n",
    "\n",
    "# Save the figure as a file (e.g., PNG)\n",
    "plt.savefig('../../Data/ClimateEducation/Figures_092024/combined_plot.svg', bbox_inches='tight')\n",
    "\n",
    "# Show the combined figure\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "occ = np.asarray(dtm.sum(axis=0)).ravel().tolist()\n",
    "counts_df = pd.DataFrame({'term': tfidf.get_feature_names_out(), 'occurrences': occ})\n",
    "counts_df.sort_values(by='occurrences', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = np.asarray(dtm.mean(axis=0)).ravel().tolist()\n",
    "weights_df = pd.DataFrame({'term': tfidf.get_feature_names_out(), 'weight': weights})\n",
    "weights_df = weights_df.sort_values(by='weight', ascending=False).head(200)\n",
    "#weights_df.to_excel('/Users/vmuccion/Documents/Projects/ClimateEducation/Output_Data/term_weight_gram12.xlsx',index = False, header=True)\n",
    "weights_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "import re\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "from nltk.tokenize import TweetTokenizer, RegexpTokenizer\n",
    "import nltk\n",
    "\n",
    "# Contraction map\n",
    "c_dict = {\n",
    "    \"ain't\": \"am not\",\n",
    "    \"aren't\": \"are not\",\n",
    "    \"can't\": \"cannot\",\n",
    "    \"can't've\": \"cannot have\",\n",
    "    \"'cause\": \"because\",\n",
    "    \"could've\": \"could have\",\n",
    "    \"couldn't\": \"could not\",\n",
    "    \"couldn't've\": \"could not have\",\n",
    "    \"didn't\": \"did not\",\n",
    "    \"doesn't\": \"does not\",\n",
    "    \"don't\": \"do not\",\n",
    "    \"hadn't\": \"had not\",\n",
    "    \"hadn't've\": \"had not have\",\n",
    "    \"hasn't\": \"has not\",\n",
    "    \"haven't\": \"have not\",\n",
    "    \"he'd\": \"he would\",\n",
    "    \"he'd've\": \"he would have\",\n",
    "    \"he'll\": \"he will\",\n",
    "    \"he'll've\": \"he will have\",\n",
    "    \"he's\": \"he is\",\n",
    "    \"how'd\": \"how did\",\n",
    "    \"how'd'y\": \"how do you\",\n",
    "    \"how'll\": \"how will\",\n",
    "    \"how's\": \"how is\",\n",
    "    \"i'd\": \"I would\",\n",
    "    \"i'd've\": \"I would have\",\n",
    "    \"i'll\": \"I will\",\n",
    "    \"i'll've\": \"I will have\",\n",
    "    \"i'm\": \"I am\",\n",
    "    \"i've\": \"I have\",\n",
    "    \"isn't\": \"is not\",\n",
    "    \"it'd\": \"it had\",\n",
    "    \"it'd've\": \"it would have\",\n",
    "    \"it'll\": \"it will\",\n",
    "    \"it'll've\": \"it will have\",\n",
    "    \"it's\": \"it is\",\n",
    "    \"let's\": \"let us\",\n",
    "    \"ma'am\": \"madam\",\n",
    "    \"mayn't\": \"may not\",\n",
    "    \"might've\": \"might have\",\n",
    "    \"mightn't\": \"might not\",\n",
    "    \"mightn't've\": \"might not have\",\n",
    "    \"must've\": \"must have\",\n",
    "    \"mustn't\": \"must not\",\n",
    "    \"mustn't've\": \"must not have\",\n",
    "    \"needn't\": \"need not\",\n",
    "    \"needn't've\": \"need not have\",\n",
    "    \"o'clock\": \"of the clock\",\n",
    "    \"oughtn't\": \"ought not\",\n",
    "    \"oughtn't've\": \"ought not have\",\n",
    "    \"shan't\": \"shall not\",\n",
    "    \"sha'n't\": \"shall not\",\n",
    "    \"shan't've\": \"shall not have\",\n",
    "    \"she'd\": \"she would\",\n",
    "    \"she'd've\": \"she would have\",\n",
    "    \"she'll\": \"she will\",\n",
    "    \"she'll've\": \"she will have\",\n",
    "    \"she's\": \"she is\",\n",
    "    \"should've\": \"should have\",\n",
    "    \"shouldn't\": \"should not\",\n",
    "    \"shouldn't've\": \"should not have\",\n",
    "    \"so've\": \"so have\",\n",
    "    \"so's\": \"so is\",\n",
    "    \"that'd\": \"that would\",\n",
    "    \"that'd've\": \"that would have\",\n",
    "    \"that's\": \"that is\",\n",
    "    \"there'd\": \"there had\",\n",
    "    \"there'd've\": \"there would have\",\n",
    "    \"there's\": \"there is\",\n",
    "    \"they'd\": \"they would\",\n",
    "    \"they'd've\": \"they would have\",\n",
    "    \"they'll\": \"they will\",\n",
    "    \"they'll've\": \"they will have\",\n",
    "    \"they're\": \"they are\",\n",
    "    \"they've\": \"they have\",\n",
    "    \"to've\": \"to have\",\n",
    "    \"wasn't\": \"was not\",\n",
    "    \"we'd\": \"we had\",\n",
    "    \"we'd've\": \"we would have\",\n",
    "    \"we'll\": \"we will\",\n",
    "    \"we'll've\": \"we will have\",\n",
    "    \"we're\": \"we are\",\n",
    "    \"we've\": \"we have\",\n",
    "    \"weren't\": \"were not\",\n",
    "    \"what'll\": \"what will\",\n",
    "    \"what'll've\": \"what will have\",\n",
    "    \"what're\": \"what are\",\n",
    "    \"what's\": \"what is\",\n",
    "    \"what've\": \"what have\",\n",
    "    \"when's\": \"when is\",\n",
    "    \"when've\": \"when have\",\n",
    "    \"where'd\": \"where did\",\n",
    "    \"where's\": \"where is\",\n",
    "    \"where've\": \"where have\",\n",
    "    \"who'll\": \"who will\",\n",
    "    \"who'll've\": \"who will have\",\n",
    "    \"who's\": \"who is\",\n",
    "    \"who've\": \"who have\",\n",
    "    \"why's\": \"why is\",\n",
    "    \"why've\": \"why have\",\n",
    "    \"will've\": \"will have\",\n",
    "    \"won't\": \"will not\",\n",
    "    \"won't've\": \"will not have\",\n",
    "    \"would've\": \"would have\",\n",
    "    \"wouldn't\": \"would not\",\n",
    "    \"wouldn't've\": \"would not have\",\n",
    "    \"y'all\": \"you all\",\n",
    "    \"y'alls\": \"you alls\",\n",
    "    \"y'all'd\": \"you all would\",\n",
    "    \"y'all'd've\": \"you all would have\",\n",
    "    \"y'all're\": \"you all are\",\n",
    "    \"y'all've\": \"you all have\",\n",
    "    \"you'd\": \"you had\",\n",
    "    \"you'd've\": \"you would have\",\n",
    "    \"you'll\": \"you you will\",\n",
    "    \"you'll've\": \"you you will have\",\n",
    "    \"you're\": \"you are\",\n",
    "    \"you've\": \"you have\"\n",
    "}\n",
    "\n",
    "# Compiling the contraction dict\n",
    "c_re = re.compile('(%s)' % '|'.join(c_dict.keys()))\n",
    "\n",
    "# List of stop words\n",
    "#add_stop = ['climate', 'change', 'education', 'science']\n",
    "#stop_words = ENGLISH_STOP_WORDS.union(add_stop)\n",
    "#nlp.Defaults.stop_words |= {\"climate\",\"change\",\"education\", \"climatic\",\"changes\",\"climat\", \"changing\", \"chang\",\n",
    "#                           \"educ\", \"educational\", \"educative\", \"teach\", \"teaching\"}\n",
    "\n",
    "stop_words = nlp.Defaults.stop_words\n",
    "\n",
    "#nlp.Defaults.stop_words.add('climate','change','education')\n",
    "\n",
    "\n",
    "# List of punctuation\n",
    "punc = list(set(string.punctuation))\n",
    "\n",
    "\n",
    "# Splits words on white spaces (leaves contractions intact) and splits out\n",
    "# trailing punctuation\n",
    "def casual_tokenizer(text):\n",
    "    tokenizer = TweetTokenizer()\n",
    "    tokens = tokenizer.tokenize(text)\n",
    "    return tokens\n",
    "\n",
    "\n",
    "def expandContractions(text, c_re=c_re):\n",
    "    def replace(match):\n",
    "        return c_dict[match.group(0)]\n",
    "    return c_re.sub(replace, text)\n",
    "\n",
    "\n",
    "def process_text(text):\n",
    "    text = casual_tokenizer(text)\n",
    "    text = [each.lower() for each in text]\n",
    "    text = [re.sub('[0-9]+', '', each) for each in text]\n",
    "    text = [expandContractions(each, c_re=c_re) for each in text]\n",
    "    text = [SnowballStemmer('english').stem(each) for each in text]\n",
    "    text = [w for w in text if w not in punc]\n",
    "    text = [w for w in text if w not in stop_words]\n",
    "    text = [each for each in text if len(each) > 1]\n",
    "    text = [each for each in text if ' ' not in each]\n",
    "    return text\n",
    "\n",
    "def top_words(topic, n_top_words):\n",
    "    return topic.argsort()[:-n_top_words - 1:-1]  \n",
    "\n",
    "\n",
    "def topic_table(model, feature_names, n_top_words):\n",
    "    topics = {}\n",
    "    for topic_idx, topic in enumerate(model.components_):\n",
    "        t = (topic_idx)\n",
    "        topics[t] = [feature_names[i] for i in top_words(topic, n_top_words)]\n",
    "    return pd.DataFrame(topics)\n",
    "\n",
    "\n",
    "def whitespace_tokenizer(text): \n",
    "    pattern = r\"(?u)\\b\\w\\w+\\b\" \n",
    "    tokenizer_regex = RegexpTokenizer(pattern)\n",
    "    tokens = tokenizer_regex.tokenize(text)\n",
    "    return tokens\n",
    "\n",
    "\n",
    "# Funtion to remove duplicate words\n",
    "def unique_words(text): \n",
    "    ulist = []\n",
    "    [ulist.append(x) for x in text if x not in ulist]\n",
    "    return ulist\n",
    "\n",
    "\n",
    "def word_count(text):\n",
    "    return len(str(text).split(' '))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Process the text\n",
    "npr_2008['processed_text'] = npr_2008['abstract_lemmatized'].apply(process_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_2008.reset_index(inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_2008.drop('level_0', inplace = True, axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_2008.drop('index', inplace = True, axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import NMF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nmf_model = NMF(n_components=15,random_state=43)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nmf_model.fit(dtm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topics_list = []\n",
    "for index,topic in enumerate(nmf_model.components_):\n",
    "    #print(f'THE TOP 10 WORDS FOR TOPIC #{index}')\n",
    "   # print([tfidf.get_feature_names()[i] for i in topic.argsort()[-5:]])\n",
    "   # print('\\n')\n",
    "    topics_list.append([tfidf.get_feature_names_out()[i] for i in topic.argsort()[-10:]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topics_df = pd.DataFrame(topics_list)\n",
    "topics_df = topics_df.T\n",
    "topics_df = topics_df.add_prefix('Topic_')\n",
    "topics_df.iloc[::-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topics_df.iloc[::-1].to_excel(\"../../Data/ClimateEducation/ListOfTopics.xlsx\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#This are the suggested title for the topics above from 0 to len(topics)\n",
    "topics = ['Science_Research','Medical_Health','Physical_Geography','Undefined_1','Sustainability_Sustainable',\n",
    "          'Adaptation_Community', 'Energy_Mitigation','Environment_Behaviour','Undefined_2','Undefined_3',\n",
    "          'Student_Learning','Disaster_Risk','Teacher_Science', 'Child_Young','Game_Gamification']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nc = len(nmf_model.components_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_df3 = pd.DataFrame(\n",
    "    nmf_model.transform(dtm), \n",
    "    columns=[\"topic_{}\".format(i) for i in range(nc)]\n",
    ").astype(float)\n",
    "topic_df3.index = npr_2008.index\n",
    "npr_topics = pd.concat([npr_2008, topic_df3], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_results = nmf_model.transform(dtm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_results.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transforming the document-term matrix\n",
    "topic_results = nmf_model.transform(dtm)\n",
    "\n",
    "# Topic mapping\n",
    "topic_mapping = {\n",
    "    0: 'Science_Research', 1: 'Medical_Health', 2: 'Physical_Geography',\n",
    "    3: 'Undefined_1', 4: 'Sustainability_Sustainable',\n",
    "    5: 'Adaptation_Community', 6: 'Energy_Mitigation',\n",
    "    7: 'Environment_Behaviour', 8: 'Undefined_2', 9: 'Undefined_3',\n",
    "    10: 'Student_Learning', 11: 'Disaster_Risk', 12: 'Teacher_Science',\n",
    "    13: 'Child_Young', 14: 'Game_Gamification'\n",
    "}\n",
    "\n",
    "# Count the number of documents for each topic\n",
    "doc_topic_counts = topic_results.argmax(axis=1)  # Get the most associated topic for each document\n",
    "topic_member_counts = np.bincount(doc_topic_counts)  # Count occurrences per topic\n",
    "\n",
    "# Create a DataFrame to map topics to their counts and labels\n",
    "topic_counts_df = pd.DataFrame({\n",
    "    'Topic': np.arange(len(topic_member_counts)),\n",
    "    'Count': topic_member_counts\n",
    "})\n",
    "\n",
    "# Add the Topic Labels to the DataFrame\n",
    "topic_counts_df['Topic Label'] = topic_counts_df['Topic'].map(topic_mapping)\n",
    "\n",
    "# Filter out topics with fewer than 100 members and sort by count in descending order\n",
    "valid_topics_df = topic_counts_df[topic_counts_df['Count'] >= 100].sort_values(by='Count', ascending=False)\n",
    "\n",
    "# Function to get the top words for each topic\n",
    "def get_top_words(model, feature_names, valid_topics, n_words=10):\n",
    "    topic_words = {}\n",
    "    for topic_idx in valid_topics:\n",
    "        top_indices = model.components_[topic_idx].argsort()[-n_words:][::-1]  # Get top word indices\n",
    "        topic_words[topic_idx] = [feature_names[i] for i in top_indices]  # Get top words\n",
    "    return topic_words\n",
    "\n",
    "# Get the top words for valid topics\n",
    "top_words = get_top_words(nmf_model, tfidf.get_feature_names_out(), valid_topics_df['Topic'].values)\n",
    "\n",
    "# Create a grid of bar plots for valid topics, ordered by document counts\n",
    "n_topics = len(top_words)\n",
    "n_words = 10  # Number of words per topic\n",
    "\n",
    "# Determine the number of rows and columns for the plots\n",
    "n_cols = 3  # Number of columns\n",
    "n_rows = int(np.ceil(n_topics / n_cols))  # Calculate rows needed\n",
    "\n",
    "# Generate a color palette with enough colors for all topics\n",
    "colors = sns.color_palette(\"tab20\", len(topic_mapping))  # Use a distinct color palette\n",
    "\n",
    "# Create a color mapping for topics based on the palette\n",
    "color_mapping = {idx: colors[i] for i, idx in enumerate(topic_mapping.keys())}\n",
    "\n",
    "fig, axs = plt.subplots(n_rows, n_cols, figsize=(15, 5 * n_rows))\n",
    "axs = axs.flatten()  # Flatten the array of axes for iteration\n",
    "\n",
    "# Plotting each topic's top words ordered by document count\n",
    "for i, (topic_idx, words) in enumerate(top_words.items()):\n",
    "    # Check if topic is in valid topics DataFrame\n",
    "    if topic_idx in valid_topics_df['Topic'].values:\n",
    "        # Create a DataFrame for the top words and their relevance\n",
    "        word_indices = np.arange(n_words)\n",
    "        word_counts = nmf_model.components_[topic_idx, np.argsort(nmf_model.components_[topic_idx])[-n_words:][::-1]]\n",
    "\n",
    "        # Create the bar plot with color from the mapping\n",
    "        axs[i].barh(word_indices, word_counts, color=color_mapping[topic_idx])  # Use the consistent color for each topic\n",
    "        axs[i].set_yticks(word_indices)\n",
    "        axs[i].set_yticklabels(words)\n",
    "\n",
    "        # Set title to include topic number and label\n",
    "        topic_label = topic_mapping.get(topic_idx, \"Unknown Topic\")  # Fetch the label\n",
    "        axs[i].set_title(f'Topic {topic_idx}: {topic_label}', fontsize=16)\n",
    "        axs[i].set_xlabel('Importance', fontsize=14)\n",
    "\n",
    "# Remove any empty subplots\n",
    "for j in range(n_topics, n_rows * n_cols):\n",
    "    fig.delaxes(axs[j])\n",
    "\n",
    "plt.tight_layout()\n",
    "#plt.savefig('../../Data/ClimateEducation/Figures_092024/Topics_words_importance.svg', format='svg', dpi='figure',\n",
    "#        bbox_inches='tight', pad_inches=0.2,\n",
    "#        facecolor='auto', edgecolor='auto',\n",
    "#        backend=None)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_counts_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.manifold import TSNE\n",
    "\n",
    "# Assuming 'topic_results' is defined as your document-topic matrix\n",
    "\n",
    "# Topic mapping\n",
    "topic_mapping = {\n",
    "    0: 'Science_Research', 1: 'Medical_Health', 2: 'Physical_Geography', \n",
    "    3: 'Undefined_1', 4: 'Sustainability_Sustainable', \n",
    "    5: 'Adaptation_Community', 6: 'Energy_Mitigation', \n",
    "    7: 'Environment_Behaviour', 8: 'Undefined_2', 9: 'Undefined_3',\n",
    "    10: 'Student_Learning', 11: 'Disaster_Risk', 12: 'Teacher_Science',\n",
    "    13: 'Child_Young', 14: 'Game_Gamification'\n",
    "}\n",
    "\n",
    "# Count number of documents for each topic\n",
    "doc_topic_counts = np.bincount(topic_results.argmax(axis=1))\n",
    "\n",
    "# Filter out topics with fewer than 100 members\n",
    "valid_topics = np.where(doc_topic_counts >= 100)[0]\n",
    "\n",
    "# Perform t-SNE on valid topic results\n",
    "# We need to select only columns corresponding to valid topics\n",
    "valid_topic_indices = np.isin(topic_results.argmax(axis=1), valid_topics)\n",
    "tsne_model = TSNE(n_components=2, random_state=0, perplexity=30, verbose=1)\n",
    "tsne_results = tsne_model.fit_transform(topic_results[valid_topic_indices])\n",
    "\n",
    "# Create a DataFrame for t-SNE results\n",
    "tsne_df = pd.DataFrame(data=tsne_results, columns=['Dim 1', 'Dim 2'])\n",
    "\n",
    "# Re-map topics to valid ones based on the original topic results\n",
    "tsne_df['Topic'] = topic_results[valid_topic_indices].argmax(axis=1)\n",
    "\n",
    "# Map topics to labels\n",
    "tsne_df['Topic Label'] = tsne_df['Topic'].map(topic_mapping)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "year = npr_2008['PubYear']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne_df['year'] = year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne_df['Topic'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne_df.to_csv('../../Data/ClimateEducation/tsne_df_output.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select the colors for each topic (ensure the order matches the topic mapping)\n",
    "colors = sns.color_palette(\"tab20\", len(topic_mapping))\n",
    "\n",
    "# Create a dictionary to map topic labels to specific colors\n",
    "color_mapping = {label: colors[i] for i, label in enumerate(topic_mapping.values())}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12, 8))\n",
    "scatter = sns.scatterplot(data=tsne_df, x='Dim 1', y='Dim 2', hue='Topic Label', palette=color_mapping, alpha=0.7)\n",
    "\n",
    "# Draw straight lines indicating the labels\n",
    "for topic in valid_topics:\n",
    "    center = tsne_df[tsne_df['Topic'] == topic][['Dim 1', 'Dim 2']].median()  # Calculate center\n",
    "    plt.text(center['Dim 1'], center['Dim 2'], topic_mapping[topic], \n",
    "             horizontalalignment='center', verticalalignment='center', fontsize=12, weight='bold')\n",
    "\n",
    "plt.title('t-SNE Visualization of Topics', fontsize=18)\n",
    "\n",
    "# Sorting the legend in alphabetical order based on the unique Topic Labels\n",
    "handles, labels = scatter.get_legend_handles_labels()\n",
    "sorted_indices = sorted(range(len(labels)), key=lambda i: labels[i])\n",
    "handles = [handles[i] for i in sorted_indices]\n",
    "labels = [labels[i] for i in sorted_indices]\n",
    "\n",
    "# Create the legend with sorted labels\n",
    "plt.legend(handles, labels, title='Topics', loc='upper center', bbox_to_anchor=(0.5, -0.08), ncol=4)\n",
    "\n",
    "# Save the figure\n",
    "plt.savefig('../../Data/ClimateEducation/Figures_092024/Climate_Education_tse.svg', format='svg', dpi='figure',\n",
    "            bbox_inches='tight', pad_inches=0.2,\n",
    "            facecolor='auto', edgecolor='auto',\n",
    "            backend=None)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_trends = tsne_df[tsne_df['year'] <= 2022]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_trends.groupby('Topic Label').sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Aggregating the data to count occurrences of each topic label per year\n",
    "topic_trends = tsne_df.groupby(['year', 'Topic Label']).size().reset_index(name='count')\n",
    "\n",
    "# Generate a color palette based on the number of unique topic labels\n",
    "unique_labels = topic_trends['Topic Label'].unique()\n",
    "num_labels = len(unique_labels)\n",
    "palette = sns.color_palette(\"tab20\", num_labels)  # Create a distinct color palette\n",
    "\n",
    "# Now we can plot the trends over time\n",
    "plt.figure(figsize=(12, 6))\n",
    "\n",
    "# Using seaborn's lineplot to show the trends with topic labels as hue\n",
    "sns.lineplot(data=topic_trends, x='year', y='count', hue='Topic Label', palette=color_mapping, marker='o')\n",
    "\n",
    "plt.title('Topic Trends Over Time', fontsize=18)\n",
    "plt.xlabel('Year', fontsize=14)\n",
    "plt.ylabel('Count of Documents', fontsize=14)\n",
    "plt.legend(title='Topics', bbox_to_anchor=(0.03, 1), loc='upper left')\n",
    "plt.xticks(rotation=45)  # Optional: Rotate x-axis labels for better readability\n",
    "plt.tight_layout()\n",
    "#plt.savefig('../../Data/ClimateEducation/Figures_092024/Topic_trends.eps', format ='eps', dpi='figure',\n",
    "#        bbox_inches='tight', pad_inches=0.2,\n",
    "#        facecolor='auto', edgecolor='auto',\n",
    "#        backend=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assuming topic_trends is already defined as your DataFrame\n",
    "# Group by 'Topic Label' and sum occurrences\n",
    "topic_sums = topic_trends.groupby('Topic Label')['count'].sum().reset_index()\n",
    "\n",
    "# Calculate the percentage of each topic\n",
    "total_count = topic_sums['count'].sum()\n",
    "topic_sums['percentage'] = (topic_sums['count'] / total_count) * 100\n",
    "\n",
    "print(topic_sums)  # View the counts and percentages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_sums['percentage'].sort_values(ascending=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Set the figure size\n",
    "plt.figure(figsize=(12, 6))\n",
    "\n",
    "topic_sums_sorted = topic_sums.sort_values(by='percentage', ascending=False)\n",
    "\n",
    "# Create a bar plot to show the percentage\n",
    "sns.barplot(data=topic_sums_sorted, x='percentage', y='Topic Label',palette=color_mapping)\n",
    "# Adding titles and labels\n",
    "plt.title('Percentage of Topics', fontsize=18)\n",
    "plt.xlabel('Percentage of Total (%)', fontsize=14)\n",
    "plt.ylabel('Topic Label', fontsize=14)\n",
    "\n",
    "# Show the plot\n",
    "plt.tight_layout()\n",
    "#plt.savefig('../../Data/ClimateEducation/Figures_092024/Topic_percentage.svg', format ='svg', dpi='figure',\n",
    "#        bbox_inches='tight', pad_inches=0.2,\n",
    "#        facecolor='auto', edgecolor='auto',\n",
    "#        backend=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a figure with two subplots: one for the bar plot and one for the line plot\n",
    "fig, ax1 = plt.subplots(2, 1, figsize=(12, 12))  # 2 rows, 1 column\n",
    "\n",
    "# **Bar Plot** (First subplot)\n",
    "#topic_sums_sorted = topic_sums.sort_values(by='percentage', ascending=False)\n",
    "sns.barplot(data=topic_sums_sorted, x='percentage', y='Topic Label', palette=color_mapping, ax=ax1[0])\n",
    "ax1[0].set_title('Percentage of Topics', fontsize=18)\n",
    "ax1[0].set_xlabel('Percentage of Total (%)', fontsize=14)\n",
    "ax1[0].set_ylabel('Topic Label', fontsize=14)\n",
    "ax1[0].tick_params(axis='y', labelsize=12)\n",
    "ax1[0].tick_params(axis='x', labelsize=12)\n",
    "\n",
    "# **Line Plot** (Second subplot)\n",
    "#topic_trends = tsne_df.groupby(['year', 'Topic Label']).size().reset_index(name='count')\n",
    "#unique_labels = topic_trends['Topic Label'].unique()\n",
    "#num_labels = len(unique_labels)\n",
    "#color_mapping = sns.color_palette(\"tab20\", num_labels)  # Create a distinct color palette\n",
    "\n",
    "sns.lineplot(data=topic_trends, x='year', y='count', hue='Topic Label', palette=color_mapping, marker='o', ax=ax1[1])\n",
    "ax1[1].set_title('Topic Trends Over Time', fontsize=18)\n",
    "ax1[1].set_xlabel('Year', fontsize=14)\n",
    "ax1[1].set_ylabel('Count of Documents', fontsize=14)\n",
    "ax1[1].legend(title='Topics', bbox_to_anchor=(0.05, 1), loc='upper left')\n",
    "ax1[1].tick_params(axis='x', rotation=45)  # Rotate x-axis labels for better readability\n",
    "ax1[1].tick_params(axis='y', labelsize=12)\n",
    "ax1[1].tick_params(axis='x', labelsize=12)\n",
    "\n",
    "# Adjust layout to prevent overlap\n",
    "plt.tight_layout()\n",
    "\n",
    "# Save the combined figure\n",
    "plt.savefig('../../Data/ClimateEducation/Figures_092024/Combined_Topic_Analysis.svg', \n",
    "            format='svg', dpi='figure',\n",
    "            bbox_inches='tight', pad_inches=0.2,\n",
    "            facecolor='auto', edgecolor='auto')\n",
    "\n",
    "# Show the combined figure\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate a color palette\n",
    "unique_labels = topic_sums['Topic Label'].unique()\n",
    "palette = sns.color_palette(\"tab20\", len(unique_labels))  # Use the same palette to maintain consistency\n",
    "\n",
    "# Create a mapping of topic labels to colors\n",
    "color_mapping = {label: palette[i] for i, label in enumerate(unique_labels)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_topic_dic = {0: 'Science_Research', 1:'Medical_Health', 2: 'Physical_Geography', \n",
    "                3: 'Undefined_1', 4: 'Sustainability_Sustainable', \n",
    "                5: 'Adaptation_Community', 6: 'Energy_Mitigation', \n",
    "                7: 'Environment_Behaviour', 8:'Undefined_2', 9:'Undefined_3',\n",
    "                10: 'Student_Learning', 11:'Disaster_Risk', 12:'Teacher_Science',\n",
    "                13: 'Child_Young', 14:'Game_Gamification'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics['Topic'] = topic_results.argmax(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics['Topic Label'] = npr_topics['Topic'].map(my_topic_dic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics[\"Topic Label\"].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics[\"Topic\"].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics[\"Topic\"].value_counts(normalize=True)*100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics[\"Topic\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_counts = npr_topics['Topic'].value_counts(normalize=True)*100\n",
    "thresholds_2 = 1\n",
    "npr_topics_select = npr_topics[npr_topics['Topic'].isin(topic_counts[topic_counts > thresholds_2].index)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics.to_csv('../../Data/ClimateEducation/npr_topics.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.colors as pltc\n",
    "import matplotlib.ticker as mtick"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics_select[\"Topic Label\"].value_counts(normalize=True)*100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics[\"Topic\"].value_counts(normalize=True)*100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = npr_topics_select.boxplot(column = ['topic_4', 'topic_10', 'topic_0', 'topic_5','topic_7', 'topic_13', 'topic_12', \n",
    "                           'topic_1','topic_6','topic_11','topic_14'])\n",
    "ax.set_xticklabels(ax.get_xticklabels(), rotation = 45, fontsize=10, ha='right', rotation_mode=\"anchor\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import pearsonr\n",
    "import matplotlib.pyplot as plt \n",
    "\n",
    "def corrfunc(x, y, hue = None, ax=None, **kws):\n",
    "    \"\"\"Plot the correlation coefficient in the top left hand corner of a plot.\"\"\"\n",
    "    r, _ = pearsonr(x, y)\n",
    "    ax = ax or plt.gca()\n",
    "    ax.annotate(f'ρ = {r:.2f}', xy=(.1, .9), xycoords=ax.transAxes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topic = npr_topics_select[['topic_10','topic_4','topic_5','topic_0','topic_7', 'topic_13', \n",
    "                      'topic_12', 'topic_1','topic_6','topic_11', 'topic_14']]\n",
    "\n",
    "npr_topic.rename(columns={'topic_10': 'Student_Learning','topic_4': 'Sustainability_Sustainable',\n",
    "                          'topic_0': 'Science_Research',\n",
    "                          'topic_5': 'Adaptation_Community',\n",
    "                          'topic_7': 'Environment_Behaviour',\n",
    "                          'topic_13': 'Child_Young', \n",
    "                          'topic_12': 'Teacher_Science',\n",
    "                          'topic_1': 'Health_Medical',\n",
    "                          'topic_6': 'Energy_Mitigation',\n",
    "                          'topic_11': 'Disaster_Risk',\n",
    "                          'topic_14': 'Game_Gamification'\n",
    "                         }, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(10, 8))\n",
    "ax = npr_topic.boxplot()\n",
    "ax.set_xticklabels(ax.get_xticklabels(), rotation = 45, fontsize=10, ha='right', rotation_mode=\"anchor\")\n",
    "plt.savefig('../../Data/ClimateEducation/Figures_092024/box_plots.svg',dpi='figure',\n",
    "        bbox_inches='tight', pad_inches=0.2,\n",
    "        facecolor='auto', edgecolor='auto',\n",
    "        backend=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = sns.boxplot(data=npr_topic)\n",
    "ax.set_xticklabels(ax.get_xticklabels(), rotation = 45, fontsize=10, ha='right', rotation_mode=\"anchor\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.style.use('default')\n",
    "\n",
    "g = sns.pairplot(\n",
    "    npr_topic,\n",
    "    x_vars=[\"Sytem_Learning_Social_Research\", \"Learning_School_Student\", \"School_Teacher_Science\",\"Environmental_Concern_Behaviour_Child\"],\n",
    "    y_vars=[\"Sytem_Learning_Social_Research\", \"Learning_School_Student\", \"School_Teacher_Science\",\"Environmental_Concern_Behaviour_Child\"],\n",
    "height=2, markers=\".\", diag_kind = 'hist')\n",
    "\n",
    "#g.map_lower(corrfunc)\n",
    "#plt.show()\n",
    "plt.savefig('../Figure_GPT2_5/Climate_Education_pairplot.eps')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Something to try will be to make a heatmap with two topic and cross check when both of them exceed a certain threshold, e.g topic university and topic children..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aa1=npr_topic.corr(method='spearman')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aa1 = aa1.sort_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aa1 = aa1.sort_index(axis =1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aa1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics['Source title']= npr_topics['Source title'].str.lower().replace('proceedings of the national academy of sciences of the united states of america','pnas')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics['Source title'] = npr_topics['Source title'].str.lower().replace('international journal of sustainability in higher education','Int Jour Sust Higher Edu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics['Source title'] = npr_topics['Source title'].str.lower().replace('international journal of environmental research and public health','IJERPH')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics['Source title'] = npr_topics['Source title'].str.lower().replace('international research in geographical and environmental education','Int. Res. Geogr. Environ.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = npr_topics.groupby(by=npr_topics['Source title'].str.lower())['Times cited'].sum()\n",
    "\n",
    "b= a/(a.sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "c = (b*100).sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(nrows=2, ncols=1, figsize =(10,15))\n",
    "x0 = c.head(20)\n",
    "x1 = (npr_topics['Source title'].str.lower().value_counts(normalize = True)*100).head(20)\n",
    "\n",
    "x0.plot.barh(ax=axes[0],color ='dimgrey')\n",
    "axes[0].set(ylabel=None)\n",
    "axes[0].set(xlabel='Proportion of citations')\n",
    "axes[0].set_xlabel('Proportion of citations', fontsize=14)\n",
    "axes[0].set_yticklabels(axes[0].get_yticklabels(), fontsize=14)\n",
    "axes[0].tick_params(axis='both', labelsize=14)\n",
    "axes[0].xaxis.set_major_formatter(mtick.PercentFormatter(decimals=1))\n",
    "#ax.set_ylim(10000, 90000)\n",
    "#ax.set_ylim(10000, 900000)\n",
    "axes[0].set_title(label = 'Top 20 journals based on citations',fontsize =16)\n",
    "\n",
    "x1.plot.barh(ax=axes[1],color ='dimgrey')\n",
    "axes[1].set(ylabel=None)\n",
    "axes[1].set(xlabel='Proportion of publications')\n",
    "axes[1].set_xlabel('Proportion of publications', fontsize=14)\n",
    "axes[1].set_yticklabels(axes[1].get_yticklabels(), fontsize=14)\n",
    "axes[1].xaxis.set_major_formatter(mtick.PercentFormatter(decimals=1))\n",
    "axes[1].tick_params(axis='both', labelsize=14)\n",
    "#ax.set_ylim(10000, 90000)\n",
    "#ax.set_ylim(10000, 900000)\n",
    "axes[1].set_title(label = 'Top 20 journals based on number of papers',fontsize =16)\n",
    "\n",
    "plt.savefig('../Figure_GPT2_8/Journals_citations_publications.eps', format = 'eps', dpi='figure',\n",
    "        bbox_inches='tight', pad_inches=0.1,\n",
    "        facecolor='auto', edgecolor='auto',\n",
    "        backend=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_2008['Source title'].str.lower().value_counts(normalize = True).head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Do you want to save the datasets?\")\n",
    "x = input()\n",
    "if x == 'yes':\n",
    "    npr_topics.to_csv('/Users/vmuccion/Documents/Projects/ClimateEducation/Output_Data/npr_2008_GPT2_October2023.csv')\n",
    "    npr_topics.to_excel('/Users/vmuccion/Documents/Projects/ClimateEducation/Output_Data/npr_2008_GPT2_October2023.xlsx',index = False, header=True)\n",
    "#    npr.to_csv('/Users/vmuccion/Documents/Projects/ClimateEducation/Data/npr.csv')\n",
    "#    npr.to_excel('/Users/vmuccion/Documents/Projects/ClimateEducation/Data/npr.xlsx',index = False, header=True)\n",
    "else:\n",
    "    print('No need to save because answer was: ' + x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Geoparsing \n",
    "## We perform now some analysis based on countries and continents. \n",
    "* First we import the geo parser for geograpghical entity recognition. We use geotext\n",
    "* We allocate the countries to the whole dataframe\n",
    "* We then assign country codes and continents to a slice of the dataframe\n",
    "* We work with this new dataframe to produce maps, spider diagrams etc..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install geotext"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Trying GeoText. It's not able to recognise when the same countries is talked more than once in the same abstract. \n",
    "#This needs to be manually implemented in the dataframe. \n",
    "from geotext import GeoText\n",
    "#places = GeoText(\"London is a great city\")\n",
    "#places.cities\n",
    "# \"London\"\n",
    "#places.country_mention\n",
    "from pycountry_convert import  country_alpha2_to_continent_code, country_alpha3_to_country_alpha2,country_name_to_country_alpha2,country_name_to_country_alpha3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Parse countries in the Abstracts\n",
    "geo_data_country = (npr_2008['Abstract']\n",
    "#      .replace(r\"\\bUSA\\b\", \"United States\", regex=True)\n",
    "       .replace((r\"\\bUSA\\b\", r\"\\bUS\\b\", r\"\\bU.S.\\b\") ,\"United States\",regex=True)      \n",
    "       .replace((r\"\\bUK\\b\", r\"\\bU.K.\\b\") ,\"United Kingdom\",regex=True)                     \n",
    "       .apply(lambda x: GeoText(x).countries)\n",
    ")\n",
    "geo_data_country.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "geo_data_country = geo_data_country.rename('Countries')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Add countries as a list to the dataframe column countries\n",
    "npr_topics['countries'] = geo_data_country"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Do you want to save the datasets?\")\n",
    "x = input()\n",
    "if x == 'yes':\n",
    "    npr_2008.to_csv('/Users/vmuccion/Documents/Projects/ClimateEducation/Output_Data/npr_2008_GPT2_Geo_October2023.csv')\n",
    "    npr_2008.to_excel('/Users/vmuccion/Documents/Projects/ClimateEducation/Output_Data/npr_2008_GPT2_Geo_October2023.xlsx',index = False, header=True)\n",
    "#    npr.to_csv('/Users/vmuccion/Documents/Projects/ClimateEducation/Data/npr.csv')\n",
    "#    npr.to_excel('/Users/vmuccion/Documents/Projects/ClimateEducation/Data/npr.xlsx',index = False, header=True)\n",
    "else:\n",
    "    print('No need to save because answer was: ' + x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Do you want to save the datasets?\")\n",
    "x = input()\n",
    "if x == 'yes':\n",
    "    npr_topics.to_csv('../../Data/ClimateEducation/Data_September_2024/npr_2008_GPT2_Geo_September2024.csv')\n",
    "    #npr_topics.to_excel('/Users/vmuccion/Documents/Projects/ClimateEducation/Output_Data/npr_topics_GPT2_Geo_October2023.xlsx',index = False, header=True)\n",
    "#    npr.to_csv('/Users/vmuccion/Documents/Projects/ClimateEducation/Data/npr.csv')\n",
    "#    npr.to_excel('/Users/vmuccion/Documents/Projects/ClimateEducation/Data/npr.xlsx',index = False, header=True)\n",
    "else:\n",
    "    print('No need to save because answer was: ' + x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.ticker as mtick"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#The result in this figure correspond to the proportion of paper mentioning a given country out of the total of the papers \n",
    "# mentioning any country. The total of the paper mentioning a country or more in its abstract is 2261 (see below). \n",
    "ax = (npr_topics['countries'].explode().reset_index().drop_duplicates(keep = \"first\")['countries'].value_counts(ascending=False, normalize=True)*100).head(33).plot.barh(figsize=(10,8))\n",
    "ax.set(xlabel=None)\n",
    "ax.set_xticklabels(ax.get_xticklabels(), fontsize=12)\n",
    "ax.set_yticklabels(ax.get_yticklabels(), fontsize=12)\n",
    "#ax.tick_params(axis='both', labelsize=12)\n",
    "ax.xaxis.set_major_formatter(mtick.PercentFormatter(decimals = 0))\n",
    "ax.grid(axis = 'y', linestyle = '--', linewidth = 0.5)\n",
    "ax.spines[\"bottom\"].set_visible(True)\n",
    "ax.spines[\"top\"].set_visible(False)\n",
    "ax.spines[\"right\"].set_visible(False)\n",
    "ax.spines[\"left\"].set_visible(True)\n",
    "\n",
    "plt.savefig('../../Data/ClimateEducation/Figures_092024/Publications_countries.eps', format = 'eps',dpi='figure',\n",
    "        bbox_inches='tight', pad_inches=0.1,\n",
    "        facecolor='auto', edgecolor='auto',\n",
    "        backend=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics['countries'].explode().reset_index().drop_duplicates(keep = \"first\")['countries'].value_counts(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = npr_topics['countries'].explode().reset_index().drop_duplicates(keep = \"first\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = test.set_index('index')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = pd.concat([test,npr_topics['Topic Label']], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = test.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = test.reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = test.drop('index',axis =1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_test = test[test.groupby('countries').countries.transform('count')>=20].copy() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_test.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_test['countries'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test['countries'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_test['countries'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_test[sub_test['Topic Label'] == 'Science_Research']['countries'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_plot = sub_test.groupby(['Topic Label', 'countries']).size().reset_index().pivot(columns='Topic Label', index='countries', values=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_plot.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_plot = df_plot.fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_plot = df_plot.iloc[:,:-3]\n",
    "#df_plot = df_plot.iloc[:,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_plot.columns = df_plot.columns.get_level_values(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_plot =df_plot.astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15,7))\n",
    "ax = sns.heatmap(df_plot.drop('Physical_Geography', axis=1).T, ax=ax, linecolor='white',annot = True,linewidths=2,cmap=\"crest\", cbar = False)\n",
    "plt.savefig('../../Data/ClimateEducation/Figures_092024/Country_Topic_association.svg', format = 'svg',dpi='figure',\n",
    "        bbox_inches='tight', pad_inches=0.1,\n",
    "        facecolor='auto', edgecolor='auto',\n",
    "        backend=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#We redo the sns heatmap plan due to a bug in Seaborne. We save the files and \n",
    "#use a different notebook to avoid having to restart this one.\n",
    "\n",
    "aa1.to_csv('../../Data/ClimateEducation/Data_September_2024/data_for_heatmap.csv')\n",
    "df_plot.to_csv('../../Data/ClimateEducation/Data_September_2024/data_for_country_counts.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#npr_new.drop_duplicates(keep = \"first\").head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#npr_new.groupby(['index', 'countries']).first()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df10 = npr_topics['countries'].explode().reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#for i, row in df10.iterrows():    \n",
    "code = []\n",
    "for p in df10.countries:\n",
    "    try:\n",
    "        aa = country_name_to_country_alpha3(p)\n",
    "        code.append(aa)\n",
    "    except:\n",
    "        code.append('')\n",
    "df10['code'] = code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#npr_2008.drop('Country_0', axis = 1, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#npr_2008['continent'].value_counts(ascending=False).head(20).plot.barh(figsize=(10,8)) #autopct='%.2f')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#npr_2008['continent'].value_counts(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics['countries'].explode().reset_index().drop_duplicates(keep = \"first\")['countries'].value_counts(ascending=False).head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics['countries'].explode().reset_index().drop_duplicates(keep = \"first\")['countries'].value_counts(ascending=False).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics['countries'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Proportion of papers mentioning specific countries\n",
    "(2261/5705)*100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df8 = npr_topics['countries'].explode().reset_index().drop_duplicates(keep = \"first\")['countries'].value_counts(ascending=False).rename_axis('countries').to_frame('counts')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr_topics['countries'].explode().reset_index().drop_duplicates(keep = \"first\")['countries']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df8.reset_index(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df8['countries'].iloc[100:157]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#for i, row in df10.iterrows():    \n",
    "code = []\n",
    "continents = []\n",
    "for p in df8.countries:\n",
    "    try:\n",
    "        aa = country_name_to_country_alpha3(p)\n",
    "        b1 = country_name_to_country_alpha2(p)\n",
    "        bb = country_alpha2_to_continent_code(b1)\n",
    "        code.append(aa)\n",
    "        continents.append(bb) \n",
    "    except:\n",
    "        code.append('')\n",
    "        continents.append('')\n",
    "df8['code'] = code\n",
    "df8['continents'] = continents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df8.to_csv(\"check_countries.csv\")  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df8['code'].iloc[56] = 'ATA'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df8['continents'].iloc[56] = 'AQ'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df8['code'].iloc[126] = 'PSE'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df8['continents'].iloc[126] = 'AS'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df8['code'].iloc[103] = 'XK'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df8['continents'].iloc[103] = 'EU'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df8.iloc[0:50,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import plotly.express as px\n",
    "import plotly.graph_objects as go\n",
    "import kaleido\n",
    "\n",
    "fig = go.Figure(data=go.Choropleth(\n",
    "    locations = df8['code'],\n",
    "    z = df8['counts'],\n",
    "    text = df8['countries'],\n",
    "    colorscale = 'sunsetdark',\n",
    "    autocolorscale=False,\n",
    "    reversescale=False,\n",
    "    #marker_line_color='darkgray',\n",
    "    marker_line_width=0.5,\n",
    "    colorbar_tickprefix = '',\n",
    "    colorbar_title = 'Country count', \n",
    "))\n",
    "fig.update_geos(projection_type=\"natural earth\", showcountries = True,showlakes = False,\n",
    "                showocean=True, oceancolor=\"LightBlue\")\n",
    "\n",
    "fig.update_layout(title_text = 'Geographical distributions of case studies')\n",
    "                                                                                                               \n",
    "fig.show()\n",
    "\n",
    "fig.write_image(\"../Figure_GPT2_8/Map_2.svg\")\n",
    "\n",
    "#plt.savefig('../Figure_GPT2_6/Map_2.eps', dpi=400)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aa = df8.groupby(df8['continents']).sum()\n",
    "ax = aa.plot.pie(subplots=True,figsize=(7,7),legend = True, autopct='%1.1f%%', labeldistance = None)\n",
    "plt.ylabel(None)\n",
    "plt.title('Studies per continent',fontsize =16, loc=\"left\")\n",
    "plt.legend(bbox_to_anchor=(1.1, 0.8), fontsize = 12)\n",
    "figure_name = 'Pie_Chart_Continents'\n",
    "plt.savefig('../Figure_GPT2_8/Pie_continents.eps', dpi='figure',\n",
    "        bbox_inches='tight', pad_inches=0.1,\n",
    "        facecolor='auto', edgecolor='auto',\n",
    "        backend=None)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aa.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_continets = {'AF': 'Africa', 'AQ':'Antarctica', 'AS': 'Asia',\n",
    "               'EU': 'Europe', 'NA':'North America', 'OC': 'Oceania','SA': 'South America'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aa.index = aa.index.map(my_continets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aa['counts'][i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a figure and axis\n",
    "fig, ax = plt.subplots(figsize=(10, 10),subplot_kw=dict(aspect=\"equal\"))\n",
    "\n",
    "wedges, texts = ax.pie(aa['counts'], wedgeprops=dict(width=0.5))\n",
    "bbox_props = dict(boxstyle=\"square,pad=0.3\", fc=\"w\", ec=\"k\", lw=0.72)\n",
    "kw = dict(arrowprops=dict(arrowstyle=\"-\"),\n",
    "          bbox=bbox_props, zorder=0, va=\"center\")\n",
    "\n",
    "for i, p in enumerate(wedges):\n",
    "    ang = (p.theta2 - p.theta1)/2. + p.theta1\n",
    "    y = np.sin(np.deg2rad(ang))\n",
    "    x = np.cos(np.deg2rad(ang))\n",
    "    horizontalalignment = {-1: \"right\", 1: \"left\"}[int(np.sign(x))]\n",
    "    connectionstyle = f\"angle,angleA=0,angleB={ang}\"\n",
    "    kw[\"arrowprops\"].update({\"connectionstyle\": connectionstyle})\n",
    "    ax.annotate((aa.index[i],aa['counts'][i]), xy=(x, y), xytext=(1.2*np.sign(x), 1*y),\n",
    "                horizontalalignment=horizontalalignment, **kw, fontsize=14)\n",
    "\n",
    "#ax.set_title(\"Matplotlib bakery: A donut\")\n",
    "figure_name = 'Pie_Chart_Continents'\n",
    "plt.savefig('../Figure_GPT2_8/Donuts_continents.eps', dpi='figure',\n",
    "        bbox_inches='tight', pad_inches=0.1,\n",
    "        facecolor='auto', edgecolor='auto',\n",
    "        backend=None)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df12 = df8.groupby(df8['continents']).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df12.reset_index(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_name = ['Africa', 'Antarctica', 'Asia', 'Europe', 'N. America', 'S. America', 'Oceania'] \n",
    "df12['Continent_Full_Name'] = full_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(6,8))\n",
    "ax = fig.add_subplot(111, projection=\"polar\")\n",
    "\n",
    "# theta has 7 different angles, and the first one repeated\n",
    "theta = np.arange(len(df12) + 1) / float(len(df12)) * 2 * np.pi\n",
    "# values has the 7 values from 'counts', with the first element repeated\n",
    "values = df12['counts'].values\n",
    "values = np.append(values, values[0])\n",
    "\n",
    "# draw the polygon and the mark the points for each angle/value combination\n",
    "l1, = ax.plot(theta, values, color=\"C2\", marker=\"o\", label=\"counts\")\n",
    "plt.xticks(theta[:-1], df12['Continent_Full_Name'], color='black', size=10)\n",
    "ax.tick_params(pad=20,labelrotation =0) # to increase the distance of the labels to the plot\n",
    "# fill the area of the polygon with green and some transparency\n",
    "#ax.fill(theta, values, 'blue', alpha=0.1)\n",
    "\n",
    "#plt.legend() # shows the legend, using the label of the line plot (useful when there is more than 1 polygon)\n",
    "#plt.title(\"Title\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df10 = npr_topics[npr_topics['Topic'] == 1]['countries'].explode().reset_index().drop_duplicates(keep = \"first\")['countries'].value_counts(ascending=False).rename_axis('countries').to_frame('counts')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "def get_keys(topic_matrix):\n",
    "    '''\n",
    "    returns an integer list of predicted topic \n",
    "    categories for a given topic matrix\n",
    "    '''\n",
    "    keys = topic_matrix.argmax(axis=1).tolist()\n",
    "    return keys\n",
    "\n",
    "def keys_to_counts(keys):\n",
    "    '''\n",
    "    returns a tuple of topic categories and their \n",
    "    accompanying magnitudes for a given list of keys\n",
    "    '''\n",
    "    count_pairs = Counter(keys).items()\n",
    "    categories = [pair[0] for pair in count_pairs]\n",
    "    counts = [pair[1] for pair in count_pairs]\n",
    "    return (categories, counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_results.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nmf_keys = get_keys(topic_results)\n",
    "nmf_categories, nmf_counts = keys_to_counts(nmf_keys)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_top_n_words(n_top_words, count_vectorizer, text_data):\n",
    "    '''\n",
    "    returns a tuple of the top n words in a sample and their \n",
    "    accompanying counts, given a CountVectorizer object and text sample\n",
    "    '''\n",
    "    vectorized_headlines = count_vectorizer.fit_transform(text_data.values)\n",
    "    vectorized_total = np.sum(vectorized_headlines, axis=0)\n",
    "    word_indices = np.flip(np.argsort(vectorized_total)[0,:], 1)\n",
    "    word_values = np.flip(np.sort(vectorized_total)[0,:],1)\n",
    "    \n",
    "    word_vectors = np.zeros((n_top_words, vectorized_headlines.shape[1]))\n",
    "    for i in range(n_top_words):\n",
    "        word_vectors[i,word_indices[0,i]] = 1\n",
    "\n",
    "    words = [word[0].encode('ascii').decode('utf-8') for \n",
    "             word in count_vectorizer.inverse_transform(word_vectors)]\n",
    "\n",
    "    return (words, word_values[0,:n_top_words].tolist()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(nmf_keys)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_topics = 15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define helper functions\n",
    "def get_top_n_words(n, keys, document_term_matrix, count_vectorizer):\n",
    "    '''\n",
    "    returns a list of n_topic strings, where each string contains the n most common \n",
    "    words in a predicted category, in order\n",
    "    '''\n",
    "    top_word_indices = []\n",
    "    for topic in range(n_topics):\n",
    "        temp_vector_sum = 0\n",
    "        for i in range(len(keys)):\n",
    "            if keys[i] == topic:\n",
    "                temp_vector_sum += document_term_matrix[i]\n",
    "        temp_vector_sum = temp_vector_sum.toarray()\n",
    "        top_n_word_indices = np.flip(np.argsort(temp_vector_sum)[0][-n:],0)\n",
    "        top_word_indices.append(top_n_word_indices)   \n",
    "    top_words = []\n",
    "    for topic in top_word_indices:\n",
    "        topic_words = []\n",
    "        for index in topic:\n",
    "            temp_word_vector = np.zeros((1,document_term_matrix.shape[1]))\n",
    "            temp_word_vector[:,index] = 1\n",
    "            the_word = count_vectorizer.inverse_transform(temp_word_vector)[0][0]\n",
    "            topic_words.append(the_word.encode('ascii','ignore').decode('utf-8'))\n",
    "        top_words.append(\" \".join(topic_words))         \n",
    "    return top_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_n_words_nmf = get_top_n_words(2, nmf_keys, dtm,tfidf)\n",
    "\n",
    "for i in range(len(top_n_words_nmf)):\n",
    "    print(\"Topic {}: \".format(i+1), top_n_words_nmf[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define helper functions\n",
    "def get_mean_topic_vectors(keys, two_dim_vectors):\n",
    "    '''\n",
    "    returns a list of centroid vectors from each predicted topic category\n",
    "    '''\n",
    "    mean_topic_vectors = []\n",
    "    for t in range(n_topics):\n",
    "        articles_in_that_topic = []\n",
    "        for i in range(len(keys)):\n",
    "            if keys[i] == t:\n",
    "                articles_in_that_topic.append(two_dim_vectors[i])    \n",
    "        \n",
    "        articles_in_that_topic = np.vstack(articles_in_that_topic)\n",
    "        mean_article_in_that_topic = np.mean(articles_in_that_topic, axis=0)\n",
    "        mean_topic_vectors.append(mean_article_in_that_topic)\n",
    "    return mean_topic_vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne_lsa_vectors_2 = tsne_lsa_model.fit_transform(topic_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nmf_mean_topic_vectors = get_mean_topic_vectors(nmf_keys, tsne_lsa_vectors_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne_lsa_vectors[:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bokeh.plotting import figure, output_file, show\n",
    "from bokeh.models import Label\n",
    "from bokeh.io import output_notebook\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "colormap = np.array([\n",
    "    \"#1f77b4\", \"#aec7e8\", \"#ff7f0e\", \"#ffbb78\", \"#2ca02c\",\n",
    "    \"#98df8a\", \"#d62728\", \"#ff9896\", \"#9467bd\", \"#c5b0d5\",\n",
    "    \"#8c564b\", \"#c49c94\", \"#e377c2\", \"#f7b6d2\", \"#7f7f7f\",\n",
    "    \"#c7c7c7\", \"#bcbd22\", \"#dbdb8d\", \"#17becf\", \"#9edae5\" ])\n",
    "colormap = colormap[:n_topics]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot = figure(title=\"t-SNE Clustering of {} NMF Topics\".format(n_topics), width=1000, height=1000)\n",
    "plot.scatter(x=tsne_lsa_vectors_2[:,0], y=tsne_lsa_vectors_2[:,1], color=colormap[nmf_keys])\n",
    "\n",
    "for t in range(n_topics):\n",
    "    label = Label(x=nmf_mean_topic_vectors[t][0], y=nmf_mean_topic_vectors[t][1], \n",
    "                  text=top_n_words_nmf[t], text_color='black', text_align='center', x_offset=0, y_offset=-8)\n",
    "    plot.add_layout(label)\n",
    "\n",
    "show(plot)\n",
    "#colormap[t]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Let's run some historical research on the overall dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr['Abstract_british'] = npr['Abstract'].apply(lambda x: britishize(x))\n",
    "npr['abstract_lemmatized'] = npr['Abstract_british'].apply(lambda row: \" \".join([w.lemma_ for w in nlp(row)]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = 'datum'\n",
    "plural = 'data'\n",
    "npr['abstract_lemmatized'] = npr['abstract_lemmatized'].str.replace(target, plural)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = 'programmeme'\n",
    "plural = 'programme'\n",
    "npr['abstract_lemmatized'] = npr['abstract_lemmatized'].str.replace(target, plural)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "npr2010 = npr[(npr['PubYear']>=1990) & (npr['PubYear'] <=2010)]\n",
    "npr2023 = npr[(npr['PubYear']>2010) & (npr['PubYear'] <=2023)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf = TfidfVectorizer(max_df=0.95, min_df=2, ngram_range=(1,2), stop_words=list(stop_words_2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def word_weights(abstracts,file_name):\n",
    "    dtm = tfidf.fit_transform(abstracts)\n",
    "    tfidf_weights = [(word, dtm.getcol(idx).sum()) for word, idx in tfidf.vocabulary_.items()]\n",
    "    feature_names = tfidf.get_feature_names_out()\n",
    "    weights = np.asarray(dtm.mean(axis=0)).ravel().tolist()\n",
    "    weights_df = pd.DataFrame({'term': tfidf.get_feature_names_out(), 'weight': weights})\n",
    "    weights_df = weights_df.sort_values(by='weight', ascending=False).head(200)\n",
    "    w = WordCloud(width=1000, height=800, mode='RGBA', background_color='white', random_state=42,colormap='viridis', max_words=500).fit_words(dict(tfidf_weights))    \n",
    "    file = weights_df.to_excel(file_name, index = False, header=True)\n",
    "    return weights_df,w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name_2010 ='../../Data/ClimateEducation/Data_September_2024/term_weight_2010_gram12.xlsx'\n",
    "weights_df_2010, w2010 = word_weights(npr2010['abstract_lemmatized'],file_name_2010)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name_2023 ='../../Data/ClimateEducation/Data_September_2024/term_weight_2023_gram12.xlsx'\n",
    "weights_df_2023,w2023 = word_weights(npr2023['abstract_lemmatized'],file_name_2023)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a figure with two subplots: one for the bar plot and one for the line plot\n",
    "fig, ax1 = plt.subplots(1, 2, figsize=(20, 18))  # 2 rows, 1 column\n",
    "\n",
    "# **Bar Plot** (First subplot)\n",
    "#topic_sums_sorted = topic_sums.sort_values(by='percentage', ascending=False)\n",
    "sns.barplot(x='weight',y = 'term',data = weights_df_2010.head(100), palette='coolwarm_r', ax=ax1[0])\n",
    "ax1[0].set_title('Word frequency 1990-2010', fontsize=18)\n",
    "ax1[0].set_xlabel('Weights', fontsize=14)\n",
    "ax1[0].set_ylabel('Words', fontsize=14)\n",
    "ax1[0].tick_params(axis='y', labelsize=12)\n",
    "ax1[0].tick_params(axis='x', labelsize=12)\n",
    "\n",
    "\n",
    "# **Bar Plot** (second subplot)\n",
    "#topic_sums_sorted = topic_sums.sort_values(by='percentage', ascending=False)\n",
    "sns.barplot(x='weight',y = 'term',data = weights_df_2023.head(100), palette='coolwarm_r', ax=ax1[1])\n",
    "ax1[1].set_title('Word frequency 2011-2023', fontsize=18)\n",
    "ax1[1].set_xlabel('Weights', fontsize=14)\n",
    "ax1[1].set_ylabel('Words', fontsize=14)\n",
    "ax1[1].tick_params(axis='y', labelsize=12)\n",
    "ax1[1].tick_params(axis='x', labelsize=12)\n",
    "\n",
    "# Adjust layout to prevent overlap\n",
    "plt.tight_layout()\n",
    "\n",
    "# Save the combined figure\n",
    "plt.savefig('../../Data/ClimateEducation/Figures_092024/WordFrequency_barplot.eps', \n",
    "            format='eps', dpi='figure',\n",
    "            bbox_inches='tight', pad_inches=0.2,\n",
    "            facecolor='auto', edgecolor='auto')\n",
    "\n",
    "# Show the combined figure\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, (ax1, ax2) = plt.subplots(nrows=1, ncols=2, figsize=(15, 8))\n",
    "csfont = {'fontname':'Arial'}\n",
    "hfont = {'fontname':'Arial'}\n",
    "\n",
    "# Plot number 1 setup\n",
    "title1 = 'Word frequency 1990-2010'\n",
    "ax1.imshow(w2010, aspect='auto')  # Use ax1 to plot the first word cloud\n",
    "ax1.set_title(title1, **csfont)  # Set title with custom font\n",
    "ax1.axis('off')  # Turn off the axis\n",
    "\n",
    "# Plot number 2 setup\n",
    "title2 = 'Word frequency 2011-2023'\n",
    "ax2.imshow(w2023, aspect='auto')  # Use ax2 to plot the second word cloud\n",
    "ax2.set_title(title2, **csfont)  # Set title for the second word cloud\n",
    "ax2.axis('off')  # Turn off the axis\n",
    "\n",
    "# Adjust the subplot layout\n",
    "plt.tight_layout(pad=5)\n",
    "\n",
    "# Save the combined figure\n",
    "plt.savefig('../../Data/ClimateEducation/Figures_092024/WordFrequency_ngrams12_time.svg', \n",
    "            format='svg', dpi='figure',\n",
    "            bbox_inches='tight', pad_inches=0.2,\n",
    "            facecolor='auto', edgecolor='auto')\n",
    "\n",
    "# Show the combined figure\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}