notebooks/working_with_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Colab preamble\n",
    "\n",
    "If this notebook is running on Google CoLab, we need to set up the required dependencies. The cell below checks if we're running on colab and tries to install the needed dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "  import google.colab\n",
    "  # Install required packages.\n",
    "  import os\n",
    "  import torch\n",
    "  os.environ['TORCH'] = torch.__version__\n",
    "  print(torch.__version__)\n",
    "\n",
    "  !pip install -q torch-scatter -f https://data.pyg.org/whl/torch-${TORCH}.html\n",
    "  !pip install -q torch-sparse -f https://data.pyg.org/whl/torch-${TORCH}.html\n",
    "  !pip install -q git+https://github.com/pyg-team/pytorch_geometric.git\n",
    "  # Other packages here\n",
    "except:\n",
    "  pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Graph Data\n",
    "\n",
    "This notebook covers fundamentals about graphs and converting data we have in some format (in this case detected cells in a `geojson` file format) to the desired format of Pytorch Geometric. For any application of Graph Neural Networks (GNNs), this is typcally one for the steps which requries much effort and where a big portion of modelling the problem comes in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example dataset\n",
    "\n",
    "While we use a very specific example in this workshop, you should hopefully be able to generalize what is learned in this notebook to any problem where you are dealing with graph-structured data. We focus on building a GNN for the problems of node and graph classification. We will use the same dataset based on cell-graphs derived from the [Prostate cANcer graDe Assessment (PANDA) Challenge](https://www.kaggle.com/competitions/prostate-cancer-grade-assessment/data). This is a dataset of about 10000 Whole Slide Images (WSI) of microscopy scans of prostate biopsy samples, accompanied with a cancer severity grading. Our main goal will be to classify each WSI according to its severity.\n",
    "\n",
    "### Pre-processing\n",
    "We have prepared the graph dataset by preprocessing the WSI images using the [CellViT](https://github.com/TIO-IKIM/CellViT) Vision Transformer. This neural network has performed cell nucleus detection on the WSI as well as cell type classification into connective, epithelial, inflammatory and neoplastic cells.\n",
    "\n",
    "See the image below for an example of how this looks in QuPath"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Example of hispathology image with annotations from CellViT](images/QuPath_cell_example.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our main goal of this workshop is to build a graph neural network which classifies these points into the given cancer grading for the dataset. While its unlikely that we'll achieve similar performance as the winners of the challange, it serves to illustrate how we can represent some very high dimensional data (the WSI in this case) in a more compact graph representation which might still be able to capture much of the important information for solving the problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Representing graphs in Pytorch Geometric\n",
    "\n",
    "Pytorch Geometric is a general purpose framework for developing neural networks which operates on different geometrical objects, primarily graphs but also point clouds. Here we will focus on how to take a datastructure in some representation, such as points in Euclidean space, and convert it into the kind of datastructures PyG expects. \n",
    "\n",
    "### The Data object\n",
    "\n",
    "In Pytorch Geometric, all objects (graphs, point clouds) are represented by a `Data` object. This object encapsulates the nodes and edges which are part of the graph. The signature of the Data constructur is as follows:\n",
    "\n",
    "```python\n",
    "class Data(x: Optional[Tensor] = None, edge_index: Optional[Tensor] = None, edge_attr: Optional[Tensor] = None, y: Optional[Tensor] = None, pos: Optional[Tensor] = None, **kwargs)\n",
    "```\n",
    "\n",
    "The parameters for the constructor are as follows:\n",
    "\n",
    " - **x** (torch.Tensor, optional) – Node feature matrix with shape [num_nodes, num_node_features]. (default: None)\n",
    " - **edge_index** (LongTensor, optional) – Graph connectivity in COO format with shape [2, num_edges]. (default: None)\n",
    " - **edge_attr** (torch.Tensor, optional) – Edge feature matrix with shape [num_edges, num_edge_features]. (default: None)\n",
    " - **y** (torch.Tensor, optional) – Graph-level or node-level ground-truth labels with arbitrary shape. (default: None)\n",
    " - **pos** (torch.Tensor, optional) – Node position matrix with shape [num_nodes, num_dimensions]. (default: None)\n",
    "\n",
    "We need to construct at least the **x** tensor which contains the node features (cell features in the case of our cell graph). The central parameter for analysing our cell graph is the **edge_index**, which contains the edge list as we've specified before. This is the first hurdle in any application of GNNs - take the graph you have in some format and convert it to the representation suitable for (in this case) Pytorch Geometric.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detected cells from CellViT\n",
    "\n",
    "In this workshop we will be working with the spatial information we've gotten from the CellViT framework. This has been used to get cell nucleus segmentation and positions with the results in `geojson` files. GeoJSON is a simple JSON-based format for encoding geographical features. The files we have from CellViT has the following structure (when represented with python datatypes):\n",
    "\n",
    "```python\n",
    "[{'type': 'Feature',\n",
    "  'id': '3cddcb72-475c-466a-aeac-bd1c3b166b71',\n",
    "  'geometry': {'type': 'MultiPoint', 'coordinates': [...]},\n",
    "  'properties': {'objectType': 'annotation',\n",
    "   'classification': {'name': 'Neoplastic', 'color': [255, 0, 0]}}},\n",
    " {'type': 'Feature',\n",
    "  'id': '12e50fb8-33e2-48da-939c-31f11694739d',\n",
    "  'geometry': {'type': 'MultiPoint', 'coordinates': [...]},\n",
    "  'properties': {'objectType': 'annotation',\n",
    "   'classification': {'name': 'Inflammatory', 'color': [34, 221, 77]}}},\n",
    " {'type': 'Feature',\n",
    "  'id': '72d03b5f-d96e-4ad9-9571-2559c70bff0c',\n",
    "  'geometry': {'type': 'MultiPoint', 'coordinates': [...]},\n",
    "  'properties': {'objectType': 'annotation',\n",
    "   'classification': {'name': 'Connective', 'color': [35, 92, 236]}}},\n",
    " {'type': 'Feature',\n",
    "  'id': 'eedec27b-d487-4ba5-89b3-578c0a214315',\n",
    "  'geometry': {'type': 'MultiPoint', 'coordinates': [...]},\n",
    "  'properties': {'objectType': 'annotation',\n",
    "   'classification': {'name': 'Dead', 'color': [254, 255, 0]}}},\n",
    " {'type': 'Feature',\n",
    "  'id': '3b297a88-7715-40a4-825c-ae4b18a4a2cf',\n",
    "  'geometry': {'type': 'MultiPoint', 'coordinates': [...]},\n",
    "  'properties': {'objectType': 'annotation',\n",
    "   'classification': {'name': 'Epithelial', 'color': [255, 159, 68]}}}]\n",
    "```\n",
    "\n",
    "As you can see, each cell type has been encapuslated in its own \"Feature\" object. Each such \"Feature\" object has an attribute called \"geometry\" and \"properties\" which we will use. The \"geometry\" attribute contains an attribute 'coordinates' which we here show as an ellipsis for compactness. In reality in contains all the coordinates of the detected cells which we will make use of to construct our cell graph.\n",
    "We want to \"transpose\" this object, so that all points (cell locations) are in the same list, while a separate list contains the classified type of each cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from pathlib import Path\n",
    "\n",
    "example_geojson_path = Path('../datasets/example/cell_detection.geojson')\n",
    "if not example_geojson_path.exists():\n",
    "    !wget https://github.com/eryl/aida-gnn-workshop-code/raw/main/datasets/example/cell_detection.geojson -P ../datasets/example \n",
    "\n",
    "with open('../datasets/example/cell_detection.geojson') as fp:\n",
    "    detected_cells_geojson = json.load(fp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def transpose_geojson(geojson_obj):\n",
    "    node_positions = []\n",
    "    node_types = []\n",
    "    for feature_obj in geojson_obj:\n",
    "        if feature_obj.get('type') == 'Feature':\n",
    "            feature_coordinates = feature_obj['geometry']['coordinates']\n",
    "            node_positions.extend(feature_coordinates)\n",
    "            feature_labels = [feature_obj['properties']['classification']['name']]*len(feature_coordinates)\n",
    "            node_types.extend(feature_labels)\n",
    "    return np.array(node_positions), node_types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "node_positions, node_types = transpose_geojson(detected_cells_geojson)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Connective', 'Dead', 'Epithelial', 'Inflammatory', 'Neoplastic'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set(node_types)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[18186.09090909,   801.1038961 ],\n",
       "       [18197.59292035,   815.59292035],\n",
       "       [18198.96638655,   834.02521008],\n",
       "       ...,\n",
       "       [20459.7340824 ,  1939.54681648],\n",
       "       [10196.14912281,  2927.88596491],\n",
       "       [11058.5270936 ,  2956.82758621]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "node_positions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset has a microns per pixel of 0.48, so if we want to constrain the radius to 20um it should be 20/0.48 pixels (since this is the coordinate system CellViT has given us the coordinates in pixel space)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To effiently build the graph, we will use the cKDTree of scipy\n",
    "import numpy as np\n",
    "from scipy.spatial import cKDTree\n",
    "\n",
    "RADIUS = 20/0.48\n",
    "def create_edges(node_positions, radius=RADIUS):\n",
    "    spatial_index = cKDTree(node_positions)\n",
    "    sparse_distances = spatial_index.sparse_distance_matrix(spatial_index, radius, output_type='coo_matrix')\n",
    "    sparse_distances.eliminate_zeros()  # We eliminate all zeros from the matrix, we don't want self-loops\n",
    "    pair_indices = np.stack([sparse_distances.row, sparse_distances.col], axis=0)\n",
    "    distances = np.copy(sparse_distances.data)\n",
    "    return pair_indices, distances\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "edges, distances = create_edges(node_positions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[14381, 14381, 14381, ...,  3967,  3967,  3982],\n",
       "       [ 3993,  3991,  3995, ...,  3969, 14335,  3979]], dtype=int32)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can inspect the edges:\n",
    "edges"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(102650,)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# And distances\n",
    "distances.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert the dataset\n",
    "\n",
    "We've included the preprocessed dataset here without any of the images. This is to keep the download size manageable since whole slide image datasets are often tens to hundreds of gigabytes in size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "def convert_geojson_to_graph(geojson_string):\n",
    "    geojson_object = json.loads(geojson_string)\n",
    "    node_positions, node_classes = transpose_geojson(geojson_object)\n",
    "    edges, distances = create_edges(node_positions)\n",
    "    return {'edges': edges, 'edge_distance': distances, 'node_positions': node_positions, 'node_types': node_classes}\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-09-04 12:57:38--  https://github.com/eryl/aida-gnn-workshop-code/releases/download/PANDa_workshop_data_v1/PANDa_dev.zip\n",
      "Resolving github.com (github.com)... 140.82.121.4\n",
      "Connecting to github.com (github.com)|140.82.121.4|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/679686685/74734d8e-ddc6-4195-a68c-325ea617de2f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230904%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230904T105741Z&X-Amz-Expires=300&X-Amz-Signature=7daafa90bc8bd216c2edb46348d6ea17a144ca5103d7e27d3d570be74b4c3d5e&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=679686685&response-content-disposition=attachment%3B%20filename%3DPANDa_dev.zip&response-content-type=application%2Foctet-stream [following]\n",
      "--2023-09-04 12:57:39--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/679686685/74734d8e-ddc6-4195-a68c-325ea617de2f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230904%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230904T105741Z&X-Amz-Expires=300&X-Amz-Signature=7daafa90bc8bd216c2edb46348d6ea17a144ca5103d7e27d3d570be74b4c3d5e&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=679686685&response-content-disposition=attachment%3B%20filename%3DPANDa_dev.zip&response-content-type=application%2Foctet-stream\n",
      "Resolving objects.githubusercontent.com (objects.githubusercontent.com)... "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
      "Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 33968577 (32M) [application/octet-stream]\n",
      "Saving to: ‘../datasets/PANDa/raw/PANDa_dev.zip.5’\n",
      "\n",
      "PANDa_dev.zip.5     100%[===================>]  32.39M  10.9MB/s    in 3.0s    \n",
      "\n",
      "2023-09-04 12:57:42 (10.9 MB/s) - ‘../datasets/PANDa/raw/PANDa_dev.zip.5’ saved [33968577/33968577]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Download the zipped dataset to ../datasets/PANDa\n",
    "!wget https://github.com/eryl/aida-gnn-workshop-code/releases/download/PANDa_workshop_data_v1/PANDa_dev.zip -P ../datasets/PANDa/raw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset is in a zip archvie, containing a csv file with all the labels and a directory called \"cell_detection.geojson\" containing the GeoJSON files. We will not extract the archive, but instead use pythons `zipfile` package to work on the files in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9c8ee5df1cd8470b9c974aea11cba335",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Creating graph data:   0%|          | 0/199 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import zipfile\n",
    "from io import StringIO, BytesIO\n",
    "import multiprocessing\n",
    "\n",
    "from tqdm.notebook import tqdm\n",
    "import pandas as pd\n",
    "\n",
    "GEOJSON_DIRECTORY = 'cell_detection.geojson'\n",
    "LABEL_COLUMN = 'isup_grade'\n",
    "ID_COLUMN = 'image_id'\n",
    "\n",
    "with zipfile.ZipFile('../datasets/PANDa/raw/PANDa_dev.zip') as zf:\n",
    "    graphs = dict()\n",
    "    csv_file = None\n",
    "    for name in tqdm(zf.namelist(), desc=\"Creating graph data\"):\n",
    "        if GEOJSON_DIRECTORY in name:\n",
    "            *_, image_id = name.split('/')\n",
    "            geojson_string = zf.read(name)\n",
    "            graph_data = convert_geojson_to_graph(geojson_string)\n",
    "            graph_data['image_id'] = image_id\n",
    "            graphs[image_id] = graph_data\n",
    "        elif '.csv' in name:\n",
    "            csv_string = zf.read(name)\n",
    "            csv_io = BytesIO(csv_string)\n",
    "            csv_file = pd.read_csv(csv_io)\n",
    "            \n",
    "    if csv_file is None:\n",
    "        raise RuntimeError(\"No csv file in archive\")\n",
    "    else:\n",
    "        for row in csv_file.to_dict('records'):\n",
    "            image_id = row[ID_COLUMN]\n",
    "            if image_id in graphs:\n",
    "                label = row[LABEL_COLUMN]\n",
    "                graphs[image_id]['label'] = label\n",
    "\n",
    "\n",
    "\n",
    "    \n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'edges': array([[ 341,  341,  342, ..., 1832, 1832, 1833],\n",
       "        [ 346,  347,  346, ..., 1831, 1833, 1832]], dtype=int32),\n",
       " 'edge_distance': array([27.0008388 , 27.0008388 , 27.0008388 , ..., 39.27425238,\n",
       "        27.90068126, 27.90068126]),\n",
       " 'node_positions': array([[ 1367.15452261,  1610.60301508],\n",
       "        [ 1367.15452261,  1610.60301508],\n",
       "        [ 2021.60902256,  5479.96992481],\n",
       "        ...,\n",
       "        [ 2482.78571429, 10642.39285714],\n",
       "        [ 2286.06796117, 10672.78640777],\n",
       "        [ 1274.47246377,  9098.9884058 ]]),\n",
       " 'node_types': ['Neoplastic',\n",
       "  'Neoplastic',\n",
       "  'Neoplastic',\n",
       "  'Neoplastic',\n",
       "  'Neoplastic',\n",
       "  'Neoplastic',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Inflammatory',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  'Connective',\n",
       "  ...],\n",
       " 'image_id': '8423df90dddd142668c18b6ab5ff7f17',\n",
       " 'label': 0}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can take a look at an arbitrary graph from the dictionary using next(iter(dict_obj.items()))\n",
    "g_id, g_dict = next(iter(graphs.items()))\n",
    "g_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Formatting the target labels\n",
    "\n",
    "Now that we've extracted the data into python dictionaries we will convert them into a Pytorch Geomtric dataset. We need to gather information about the categorical variables we have: our cell types and our labels. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0, 1: 1, 2: 2, 3: 3, 4: 4, 5: 5}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can inspect what labels we have by putting them in a set\n",
    "labels = set()\n",
    "for graph in graphs.values():\n",
    "    labels.add(graph['label'])\n",
    "label_map = {node_type: i for i,node_type in enumerate(sorted(labels))}\n",
    "label_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the labels already are integers from 0-5, so the label_map is realy not needed, but in general we might wan't to support e.g. string labels in which case the above label map is needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding node types\n",
    "The cell detection framework has also classified the cells into different types. We will need to encode these as integer values for the neural networks. We start by creating a mapping from the string-valued types to a integer one. Its preferable if this mapping is stable, which is the reason why we sort the string values  before creating the mapping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Connective': 0,\n",
       " 'Dead': 1,\n",
       " 'Epithelial': 2,\n",
       " 'Inflammatory': 3,\n",
       " 'Neoplastic': 4}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "node_types = set()\n",
    "for graph in graphs.values():\n",
    "    node_types.update(graph['node_types'])\n",
    "node_type_map = {node_type: i for i,node_type in enumerate(sorted(node_types))}\n",
    "node_type_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now go through all the graphs and convert arbitrary node types and labels to integer tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "for graph in graphs.values():\n",
    "    node_types = graph['node_types']\n",
    "    node_types = torch.tensor([node_type_map[node_type] for node_type in node_types])\n",
    "    graph['node_types'] = node_types\n",
    "    label = graph['label']\n",
    "    label = torch.tensor(label_map[label])\n",
    "    graph['label'] = label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have everything we need to create the `Data` objects for Pytorch Geometric, recall the signature of this class:\n",
    "\n",
    "```python\n",
    "class Data(x: Optional[Tensor] = None, edge_index: Optional[Tensor] = None, edge_attr: Optional[Tensor] = None, y: Optional[Tensor] = None, pos: Optional[Tensor] = None, **kwargs)\n",
    "```\n",
    "\n",
    "The parameters for the constructor are as follows:\n",
    "\n",
    " - **x** (torch.Tensor, optional) – Node feature matrix with shape [num_nodes, num_node_features]. (default: None)\n",
    " - **edge_index** (LongTensor, optional) – Graph connectivity in COO format with shape [2, num_edges]. (default: None)\n",
    " - **edge_attr** (torch.Tensor, optional) – Edge feature matrix with shape [num_edges, num_edge_features]. (default: None)\n",
    " - **y** (torch.Tensor, optional) – Graph-level or node-level ground-truth labels with arbitrary shape. (default: None)\n",
    " - **pos** (torch.Tensor, optional) – Node position matrix with shape [num_nodes, num_dimensions]. (default: None)\n",
    "\n",
    "We will use the following mapping from the graph dictionaries we have created:\n",
    " - **x** = `graph['node_types']`\n",
    " - **edge_index** = `graph['edges']`\n",
    " - **edge_attr** = `graph['edge_distances']`\n",
    " - **y** = `graph['label']`\n",
    " - **pos** = `graph['node_positions']`\n",
    " \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch_geometric.data import Data\n",
    "\n",
    "def create_pyg_data(graph_dict):\n",
    "    x=graph_dict['node_types']\n",
    "    edge_index=graph_dict['edges']\n",
    "    edge_attr=graph_dict['edge_distance']\n",
    "    y=graph_dict['label']\n",
    "    pos=graph_dict['node_positions']\n",
    "    \n",
    "    return Data(x=x, edge_index=edge_index, edge_attr=edge_attr, y=y, pos=pos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Data(x=[1839], edge_index=[2, 2748], edge_attr=[2748], y=0, pos=[1839, 2])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyg_data = [create_pyg_data(g) for g in graphs.values()]\n",
    "pyg_data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pytorch Geometric Dataset\n",
    "\n",
    "While pytorch support using a list of `Data` objects for input to its specialized DataLoader, we can get convenient features such as caching of the processed results by using the `Dataset` class in Pytorch Geometric. This class has an interface with some functions which we will overwrite with the code we defined above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import zipfile\n",
    "from io import StringIO, BytesIO\n",
    "import multiprocessing\n",
    "from collections import Counter\n",
    "from typing import Sequence, Literal, Optional\n",
    "import copy\n",
    "import json\n",
    "\n",
    "from torch_geometric.data import InMemoryDataset, download_url, Data\n",
    "from torch_geometric.data.dataset import IndexType\n",
    "from tqdm.notebook import tqdm\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy.spatial import cKDTree\n",
    "\n",
    "\n",
    "GEOJSON_DIRECTORY = 'cell_detection.geojson'\n",
    "LABEL_COLUMN = 'isup_grade'\n",
    "ID_COLUMN = 'image_id'\n",
    "MPP = 0.48\n",
    "RADIUS_MICRONS = 20\n",
    "RADIUS = RADIUS_MICRONS / MPP\n",
    "\n",
    "PANDA_FULL_URL = 'https://github.com/eryl/aida-gnn-workshop-code/releases/download/PANDa_workshop_data_full_v1/PANDa_{}.zip'\n",
    "PANDA_SUBSET_URL = 'https://github.com/eryl/aida-gnn-workshop-code/releases/download/PANDa_workshop_data_v1/PANDa_{}.zip'\n",
    "\n",
    "# You can switch to using the full 10615 files by setting this to True. \n",
    "# Warning! It will require a lot of memory to hold it in RAM. \n",
    "# The total download is about 1.5GB for the full set and 300MB for the subset.\n",
    "FULL_DATASET = False\n",
    "if FULL_DATASET:\n",
    "    DATASET_URL = PANDA_FULL_URL\n",
    "    DATASET_DIR = '../datasets/PANDa_full'\n",
    "else:\n",
    "    DATASET_URL = PANDA_SUBSET_URL\n",
    "    DATASET_DIR = '../datasets/PANDa'\n",
    "\n",
    "\n",
    "class PANDaGraphDataset(InMemoryDataset):\n",
    "    data_url = DATASET_URL\n",
    "    data_split = ''\n",
    "    \n",
    "    def __init__(self, root=DATASET_DIR, transform=None, pre_transform=None, pre_filter=None):\n",
    "        super().__init__(root, transform, pre_transform, pre_filter)\n",
    "        self.data, self.slices, self.label_map, self.node_type_map = torch.load(self.processed_paths[0])\n",
    "    \n",
    "    @property\n",
    "    def raw_file_names(self):\n",
    "        return [f'PANDa_{self.data_split}.zip']\n",
    "\n",
    "    @property\n",
    "    def processed_file_names(self):\n",
    "        return [f'PANDa_{self.data_split}.pt']\n",
    "\n",
    "    def download(self):\n",
    "        # Download to `self.raw_dir`.\n",
    "        url = self.data_url.format(self.data_split)\n",
    "        download_url(url, self.raw_dir)\n",
    "\n",
    "    def _transpose_geojson(self, geojson_obj):\n",
    "        node_positions = []\n",
    "        node_types = []\n",
    "        for feature_obj in geojson_obj:\n",
    "            if feature_obj.get('type') == 'Feature':\n",
    "                feature_coordinates = feature_obj['geometry']['coordinates']\n",
    "                node_positions.extend(feature_coordinates)\n",
    "                feature_labels = [feature_obj['properties']['classification']['name']]*len(feature_coordinates)\n",
    "                node_types.extend(feature_labels)\n",
    "        return np.array(node_positions), node_types\n",
    "\n",
    "    def _create_edges(self, node_positions, radius=RADIUS):\n",
    "        spatial_index = cKDTree(node_positions)\n",
    "        sparse_distances = spatial_index.sparse_distance_matrix(spatial_index, radius, output_type='coo_matrix')\n",
    "        sparse_distances.eliminate_zeros()  # We eliminate all zeros from the matrix, we don't want self-loops\n",
    "        pair_indices = np.stack([sparse_distances.row, sparse_distances.col], axis=0)\n",
    "        distances = np.copy(sparse_distances.data)\n",
    "        return pair_indices, distances\n",
    "\n",
    "    def _convert_geojson_to_graph(self, geojson_string):\n",
    "        geojson_object = json.loads(geojson_string)\n",
    "        node_positions, node_classes = self._transpose_geojson(geojson_object)\n",
    "        edges, distances = self._create_edges(node_positions)\n",
    "        return {'edges': edges, 'edge_distance': distances, 'node_positions': node_positions, 'node_types': node_classes}\n",
    "\n",
    "    def _read_geojson_archive(self):\n",
    "        LABEL_COLUMN = 'isup_grade'\n",
    "        ID_COLUMN = 'image_id'  \n",
    "        graphs = dict()\n",
    "\n",
    "        for filename in self.raw_file_names:\n",
    "            file_path = Path(self.raw_dir) / filename\n",
    "            with zipfile.ZipFile(file_path) as zf:\n",
    "                graphs = dict()\n",
    "                csv_file = None\n",
    "                for name in tqdm(zf.namelist(), desc=\"Creating graph data\"):\n",
    "                    if GEOJSON_DIRECTORY in name:\n",
    "                        *_, image_id = name.split('/')\n",
    "                        geojson_string = zf.read(name)\n",
    "                        graph_data = self._convert_geojson_to_graph(geojson_string)\n",
    "                        graph_data['image_id'] = image_id\n",
    "                        graphs[image_id] = graph_data\n",
    "                    elif '.csv' in name:\n",
    "                        csv_string = zf.read(name)\n",
    "                        csv_io = BytesIO(csv_string)\n",
    "                        csv_file = pd.read_csv(csv_io)\n",
    "                        \n",
    "                if csv_file is None:\n",
    "                    raise RuntimeError(\"No csv file in archive\")\n",
    "                else:\n",
    "                    for row in csv_file.to_dict('records'):\n",
    "                        image_id = row[ID_COLUMN]\n",
    "                        if image_id in graphs:\n",
    "                            label = row[LABEL_COLUMN]\n",
    "                            graphs[image_id]['label'] = label \n",
    "        return graphs\n",
    "    \n",
    "    def _convert_labels_inplace(self, graphs):\n",
    "        labels = set()\n",
    "        for graph in graphs.values():\n",
    "            labels.add(graph['label'])\n",
    "        label_map = {node_type: i for i,node_type in enumerate(sorted(labels))}\n",
    "        for graph in graphs.values():\n",
    "            label = graph['label']\n",
    "            encoded_label = torch.tensor(label_map[label])\n",
    "            graph['label'] = encoded_label\n",
    "        return label_map\n",
    "\n",
    "    def _convert_node_types_inplace(self, graphs):\n",
    "        node_types = set()\n",
    "        for graph in graphs.values():\n",
    "            node_types.update(graph['node_types'])\n",
    "        node_type_map = {node_type: i for i,node_type in enumerate(sorted(node_types))}\n",
    "        for graph in graphs.values():\n",
    "            node_types = graph['node_types']\n",
    "            node_types = torch.tensor([node_type_map[node_type] for node_type in node_types])\n",
    "            graph['node_types'] = node_types\n",
    "        return node_type_map\n",
    "\n",
    "    def _convert_to_pyg_data(self, graph):\n",
    "        x=graph['node_types']\n",
    "        edge_index=torch.tensor(graph['edges'])\n",
    "        edge_attr=torch.tensor(graph['edge_distance'])\n",
    "        y=graph['label']\n",
    "        pos=torch.tensor(graph['node_positions'])\n",
    "        return Data(x=x, edge_index=edge_index, edge_attr=edge_attr, y=y, pos=pos)\n",
    "\n",
    "    def process(self):\n",
    "        # Read data huge `Data` list.\n",
    "        graphs = self._read_geojson_archive()\n",
    "        label_map = self._convert_labels_inplace(graphs)\n",
    "        node_type_map = self._convert_node_types_inplace(graphs)\n",
    "\n",
    "        data_list = [self._convert_to_pyg_data(graph) for graph in tqdm(graphs.values(), desc=\"Converting graphs\")]\n",
    "\n",
    "        if self.pre_filter is not None:\n",
    "            data_list = [data for data in data_list if self.pre_filter(data)]\n",
    "\n",
    "        if self.pre_transform is not None:\n",
    "            data_list = [self.pre_transform(data) for data in data_list]\n",
    "\n",
    "        data, slices = self.collate(data_list)\n",
    "        torch.save((data, slices, label_map, node_type_map), self.processed_paths[0])\n",
    "\n",
    "    \n",
    "class PANDaGraphDatasetTrain(PANDaGraphDataset):\n",
    "    data_split = 'train'\n",
    "\n",
    "class PANDaGraphDatasetDev(PANDaGraphDataset):\n",
    "    data_split = 'dev'\n",
    "\n",
    "class PANDaGraphDatasetTest(PANDaGraphDataset):\n",
    "    data_split = 'test'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset = PANDaGraphDatasetTrain()\n",
    "test_dataset = PANDaGraphDatasetTest()\n",
    "dev_dataset = PANDaGraphDatasetDev()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Data(x=[4195], edge_index=[2, 42532], edge_attr=[42532], y=[1], pos=[4195, 2])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dataset[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataLoaders\n",
    "\n",
    "Pytorch Geometric has its own dataloaders. It allows for efficient batching of graphs by using the fact that all of the PyG operations rely on sparse implementations. By taking all graphs and essentially putting them into on large batch graph, the algorithms dont have to keep special track of batch examples as long as the operations are restricted to follow the graph structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch_geometric.loader import DataLoader\n",
    "batch_size = 32\n",
    "train_dataloader = DataLoader(train_dataset, batch_size=batch_size, drop_last=False, shuffle=True, num_workers=4)\n",
    "dev_dataloader = DataLoader(dev_dataset, batch_size=batch_size, drop_last=False, shuffle=False, num_workers=4)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DataBatch(x=[465584], edge_index=[2, 3001512], edge_attr=[3001512], y=[32], pos=[465584, 2], batch=[465584], ptr=[33])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inspect a batch from the training dataset\n",
    "batch = next(iter(train_dataloader))\n",
    "batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pytorch Geometric implements the batches with the assumptions that we will be using sparse operations on them. Instead of creating one tensor (e.g. for the node attributes), it creates one large tensor which packs _all_ of the graphs in the batch. If we think of this from the perspective of an adjacency graph, it essentially greates a block-diagonal adjacency matrix.\n",
    "$$\n",
    "A = \\begin{bmatrix}\n",
    "\\mathbf{A}_1& \\mathbf{0} & \\dots & \\mathbf{0}\\\\\n",
    "\\mathbf{0}& \\mathbf{A}_2 & \\dots & \\mathbf{0}\\\\\n",
    "\\vdots & \\vdots & \\ddots & \\mathbf{0} \\\\\n",
    "\\mathbf{0}& \\mathbf{0} & \\dots & \\mathbf{A}_n\\\\\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "This has the advantage that if the graph convolution operations only aggregates information based on the graph neighbourhood, no special case has to be programmed for batches vs. non-batched inputs which simplifies the implementation.\n",
    "The batch object we get from the dataloader has a special `batch` attribute which we can use to determine which graph in the batch a node belongs to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect a batch from the training dataset\n",
    "batch.batch\n",
    "# Picking out the node features of the first graph\n",
    "first_node_features = batch.x[batch.batch == 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The batch object implements an easy to use interface to get the individual graphs by just subscripting the batch object with the graph index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(True)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.all(batch[0].x == first_node_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusions\n",
    "\n",
    "We've now created a Pytorch Geometric dataset which we can easily integrate into the existing PyG models. In the next notebook we will look at how we do that by designing our own Graph Neural Networks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "aida_workshop_gnn_v2",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}