Part 5 - Network Motifs with PySpark and GraphFrames.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7530f266-3e21-4c6e-acfe-663bc846b059",
   "metadata": {},
   "source": [
    "This is part 5 of a course called Full Stack Graph Machine Learning by <a href=\"https://linkedin.com/in/russelljurney\">Russell Jurney</a>. For team training inquiries contact <a href=\"mailto:rjurney@graphlet.ai\">rjurney@graphlet.ai</a>.\n",
    "\n",
    "This code is still in [graphml_class/stats](graphml_class/stats) but is being converted into a GraphFrames motif finding tutorial. [Follow this issue](https://github.com/graphframes/graphframes/issues/493) for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92f97d5a-9118-4331-adc3-4c3266eca84d",
   "metadata": {},
   "source": [
    "# DISCLAIMER:\n",
    "\n",
    "* This code resides in the folder graphml/stats/\n",
    "* It will not currently run in a notebook or a docker container. I haven't figured out why, but graph.py takes forever.\n",
    "* You need to run the code on a computer directly\n",
    "* The runtime order is: [graphml_class/stats/download.py](graphml_class/stats/download.py) --> [graphml_class/stats/xml_to_parquet.py](graphml_class/stats/xml_to_parquet.py) --> [graphml_class/stats/graph.py](graphml_class/stats/graph.py) --> [graphml_class/stats/motif.py](graphml_class/stats/motif.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1be3258d-a06a-4334-aa2b-f955588ec80f",
   "metadata": {},
   "source": [
    "# Network Motifs with PySpark and GraphFrames\n",
    "\n",
    "I'll be honest with you... I am a big data, large knowledge graph specialist. All the data we have used so far is very small to me. While it is difficult to cover scalable methods for everything, I wanted to introduce you to a tool for PySpark called [GraphFrames](https://graphframes.github.io/graphframes/docs/_site/index.html). It has some powerful utility methods like [connected components](https://graphframes.github.io/graphframes/docs/_site/user-guide.html#connected-components) that many teams use to perform tasks like merging nodes during entity resolution - node deduplication. It can also perform property graph [motif finding](https://graphframes.github.io/graphframes/docs/_site/user-guide.html#motif-finding) for networks with billions of nodes and edges.\n",
    "\n",
    "GraphFrames are graphs [created from node / edge lists](https://graphframes.github.io/graphframes/docs/_site/user-guide.html#creating-graphframes) which are [pyspark.sql.DataFrames](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/dataframe.html). This means you get the power of PySpark to write arbitrary graph operations on large datasets thanks to [Spark](https://spark.apache.org/docs/latest/).\n",
    "\n",
    "## Imperative vs Declarative\n",
    "\n",
    "SQL is a declarative language. PySpark is an imperative API, with a declarative SQL interface if you prefer. We'll be using the `pyspark.sql.DataFrame` and `pyspark.sql.functions` APIs to process data step-by-step. This is different from Pandas, which is fairly declarative in its APIs. Keep that in mind below... we get to carefully control how the dataflow works, which can be time consuming compared to a single Pandas command that does more than what Spark `DataFrames` do - MapReduce and iterate :)\n",
    "\n",
    "## Running Checks\n",
    "\n",
    "It is easy to mess up an algorithm when building knowledge graphs in PySpark / GraphFrames. You will notice that throughout  the script below, I check and print counts as I go. I encourage you to do this as well, or your scripts will seem to run but will produce bad knowledge graphs that produce bad answers.\n",
    "\n",
    "## Running GraphFrames\n",
    "\n",
    "To run GraphFrames locally from a shell, you can import the package with the `spark-shell` or `pyspark` `--packages` argument:\n",
    "\n",
    "```bash\n",
    "pyspark --packages graphframes:graphframes:0.8.3-spark3.5-s_2.12\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5ce6c89-11dc-40de-bc27-94f3089068fb",
   "metadata": {},
   "source": [
    "### Real-World Network Motifs and Higher-Order Networks\n",
    "\n",
    "Remember these slides? This is a real-world use of network motif matching to find sources of risk for banks in terms of money laundering and terrorism funding occurring in financial networks of people and companies.\n",
    "\n",
    "<center><img src=\"images/Multiple-Path-Indirect-Ownership-Motif.jpg\" width=\"800px\" /></center>\n",
    "<center>A well known pattern to hide Ultimate Beneficial Ownership (UBO) of a company</center>\n",
    "\n",
    "<br />\n",
    "\n",
    "<center><img src=\"images/PySpark-GraphFrames-Motif-Search-Python-Code.jpg\" width=\"800px\" /></center>\n",
    "<center>The PySpark / GraphFrames pseudo-code that implements [most of] the motif above</center>\n",
    "\n",
    "<br />\n",
    "\n",
    "The next motif is more complicated... it uses Sentence Transformers with PySpark - [[here's how](https://stackoverflow.com/questions/72398129/creating-a-sentence-transformer-model-in-spark-mllib)] to perform what I think is a new type of network motif - a _semantic network motif_.\n",
    "\n",
    "<center><img src=\"images/Corrupt-Incorporation-Services-Motif.jpg\" width=\"800px\" /></center>\n",
    "<center>A more complex <i>semantic network motif</i> using Sentence Transformers to do fuzzy string matching of officer names</center>\n",
    "\n",
    "<br />\n",
    "\n",
    "While I don't have time to demonstrate it at present, once you have network motifs, it is possible to group by all members of a motif match and combine their edges into a new, higher-order node with new semantics. If you link or [cluster](https://arxiv.org/abs/1612.08447) these new nodes, they form a [higher-order network](https://arxiv.org/abs/2104.11329). If the motif is a pattern important to your problem domain, nodes in this higher-order network might exist in a _solution space_ rather than your graph's original _problem space_. For example, if you define risk motifs in a financial network, the way they cluster can show you centers of risk. Pattern matching is a different approach than graph machine learning but can yield similarly powerful results.\n",
    "\n",
    "<center><img src=\"images/Higher-Order-Networks-Using-Edge-Projection-via-Property-Graphlet-Minors.jpg\" width=\"800px\" /></center>\n",
    "<center>Forming a higher-order network using network motif clustering</center>\n",
    "\n",
    "<br />\n",
    "\n",
    "For more information, check out Stanfor SNAP's page on [Higher-order organization of complex networks](https://snap.stanford.edu/higher-order/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecc5122f-bcbc-4c20-ba3a-b4d01fd0b56a",
   "metadata": {},
   "source": [
    "## Hold up! This class isn't a notebook yet...\n",
    "\n",
    "This class still exists as code in the folder [graphml_class/stats/](graphml_class/stats/) while I convert it into a [network motif finding tutorial](https://github.com/graphframes/graphframes/tree/rjurney/motif-tutorial/python/graphframes/tutorials) for the [GraphFrames project](https://graphframes.github.io/graphframes/docs/_site/index.html). [Follow this issue](https://github.com/graphframes/graphframes/issues/493) to find the class when it is finished."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bc587a8-fd3c-4f2a-b49b-f714dcd0f2a7",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}