notebooks/das-users-guide.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b1245b5f",
   "metadata": {},
   "source": [
    "# DAS - User's Guide\n",
    "\n",
    "Atomspace is the hypergraph OpenCog Hyperon uses to represent and store knowledge, being the source of knowledge for AI agents and the container of any computational result that might be created or achieved during their execution.\n",
    "\n",
    "The __Distributed Atomspace (DAS)__ is an extension of OpenCog Hyperon's Atomspace into a more independent component designed to support multiple simultaneous connections with different AI algorithms, providing a flexible query interface to distributed knowledge bases. It can be used as a component (e.g. a Python library) or as a stand-alone server to store essentially arbitrarily large knowledge bases and provide means for the agents to traverse regions of the hypergraphs and perform global queries involving properties, connectivity, subgraph topology, etc.\n",
    "\n",
    "Regardless of being used locally or remotely, DAS provides the exact same API to query or traverse the Atomspace. This API is fully documented [here](https://singnet.github.io/das-query-engine/api/das/). In this document we provide examples and best practices to use this API with each type of DAS."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85a937b5",
   "metadata": {},
   "source": [
    "## Table of contents\n",
    "\n",
    "* [Local DAS with data in RAM](#ramdas)\n",
    "    * [Adding atoms](#addatoms)\n",
    "    * [Fetching from a DAS server](#fetch)\n",
    "    * [Getting atoms by their properties](#atomquery)\n",
    "    * [Traversing the hypergraph](#traversing)\n",
    "    * [Pattern Matcher Queries](#patternmatcher)\n",
    "* [Connecting to a remote DAS](#remotedas)\n",
    "    * [Querying a remote DAS](#remotequery)\n",
    "    * [Custom indexes](#customindex)\n",
    "* [Starting a DAS Server](#dasserver)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3a64bb1",
   "metadata": {},
   "source": [
    "<a id='ramdas'></a>\n",
    "## Local DAS with data in RAM\n",
    "\n",
    "\n",
    "A local DAS stores atoms as Python dicts in RAM. One can create an empty DAS by calling the basic constructor with no parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "52c5e547",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0, 0)\n"
     ]
    }
   ],
   "source": [
    "from hyperon_das import DistributedAtomSpace\n",
    "\n",
    "das = DistributedAtomSpace()\n",
    "print(das.count_atoms())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f324d460",
   "metadata": {},
   "source": [
    "This is equivalent to instantiating it passing `query_engine='local'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "47fe108a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0, 0)\n"
     ]
    }
   ],
   "source": [
    "from hyperon_das import DistributedAtomSpace\n",
    "\n",
    "das = DistributedAtomSpace(query_engine='local')\n",
    "print(das.count_atoms())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a9e5f79",
   "metadata": {},
   "source": [
    "The `query_engine` parameter is used to select from `local` or `remote` DAS. [Remote DAS](#remotedas) is explained later in this document.\n",
    "\n",
    "A local DAS can be populated using the methods `add_node()` and `add_link()`. Let's use this simple knowledge base as an example."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d7739be",
   "metadata": {},
   "source": [
    "<p align=\"center\">\n",
    "<img src=\"../docs/assets/pmquery_1.png\" width=\"400\"/>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1fb5b50",
   "metadata": {},
   "source": [
    "We have only one type of node (e.g. Concept) to represent animals and two types of links, (e.g. Inheritance and Similarity) to represent relations between them."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcba0a0d",
   "metadata": {},
   "source": [
    "<a id='addatoms'></a>\n",
    "### Adding atoms\n",
    "\n",
    "We can add nodes explicitly by calling `add_node()` passing a Python dict representing the node. This dict may contain any number of keys associated to values of any type (including lists, sets, nested dicts, etc) , which are all recorded with the node, but must contain at least the keys `type` and `name` mapping to strings which define the node uniquely, i.e. two nodes with the same `type` and `name` are considered to be the same entity.\n",
    "\n",
    "`add_link()` works mostly in the same way. For links, the mandatory fields are `type` and `targets`, which is a list of Python dicts representing either nodes or nested links."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9159075d",
   "metadata": {},
   "outputs": [],
   "source": [
    "    das.add_node({\"type\": \"Concept\", \"name\": \"human\"})\n",
    "    das.add_node({\"type\": \"Concept\", \"name\": \"monkey\"})\n",
    "    das.add_node({\"type\": \"Concept\", \"name\": \"chimp\"})\n",
    "    das.add_node({\"type\": \"Concept\", \"name\": \"mammal\"})\n",
    "    das.add_node({\"type\": \"Concept\", \"name\": \"reptile\"})\n",
    "    das.add_node({\"type\": \"Concept\", \"name\": \"snake\"})\n",
    "    das.add_node({\"type\": \"Concept\", \"name\": \"dinosaur\"})\n",
    "    das.add_node({\"type\": \"Concept\", \"name\": \"triceratops\"})\n",
    "    das.add_node({\"type\": \"Concept\", \"name\": \"earthworm\"})\n",
    "    das.add_node({\"type\": \"Concept\", \"name\": \"rhino\"})\n",
    "    das.add_node({\"type\": \"Concept\", \"name\": \"vine\"})\n",
    "    das.add_node({\"type\": \"Concept\", \"name\": \"ent\"})\n",
    "    das.add_node({\"type\": \"Concept\", \"name\": \"animal\"})\n",
    "    das.add_node({\"type\": \"Concept\", \"name\": \"plant\"}) ;"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8896f60f",
   "metadata": {},
   "source": [
    "We can also add nodes implicitly while adding links."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "32885b0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "das.add_link(\n",
    "    {\n",
    "        \"type\": \"Similarity\",\n",
    "        \"targets\": [\n",
    "            {\"type\": \"Concept\", \"name\": \"human\"},\n",
    "            {\"type\": \"Concept\", \"name\": \"monkey\"},\n",
    "        ],\n",
    "    }\n",
    ") ;"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fd74e79",
   "metadata": {},
   "source": [
    "\"human\" and \"monkey\" would be inserted if they hadn't been inserted before. Adding the node or link more than once is allowed and has no side effects. So let's add the whole set of links from our knowledge base."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cb36541d",
   "metadata": {},
   "outputs": [],
   "source": [
    "das.add_link( { \"type\": \"Similarity\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"human\"}, {\"type\": \"Concept\", \"name\": \"monkey\"}, ], })\n",
    "das.add_link( { \"type\": \"Similarity\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"human\"}, {\"type\": \"Concept\", \"name\": \"chimp\"}, ], })\n",
    "das.add_link( { \"type\": \"Similarity\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"chimp\"}, {\"type\": \"Concept\", \"name\": \"monkey\"}, ], })\n",
    "das.add_link( { \"type\": \"Similarity\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"snake\"}, {\"type\": \"Concept\", \"name\": \"earthworm\"}, ], })\n",
    "das.add_link( { \"type\": \"Similarity\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"rhino\"}, {\"type\": \"Concept\", \"name\": \"triceratops\"}, ], })\n",
    "das.add_link( { \"type\": \"Similarity\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"snake\"}, {\"type\": \"Concept\", \"name\": \"vine\"}, ], })\n",
    "das.add_link( { \"type\": \"Similarity\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"human\"}, {\"type\": \"Concept\", \"name\": \"ent\"}, ], })\n",
    "das.add_link( { \"type\": \"Similarity\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"monkey\"}, {\"type\": \"Concept\", \"name\": \"human\"}, ], })\n",
    "das.add_link( { \"type\": \"Similarity\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"chimp\"}, {\"type\": \"Concept\", \"name\": \"human\"}, ], })\n",
    "das.add_link( { \"type\": \"Similarity\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"monkey\"}, {\"type\": \"Concept\", \"name\": \"chimp\"}, ], })\n",
    "das.add_link( { \"type\": \"Similarity\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"earthworm\"}, {\"type\": \"Concept\", \"name\": \"snake\"}, ], })\n",
    "das.add_link( { \"type\": \"Similarity\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"triceratops\"}, {\"type\": \"Concept\", \"name\": \"rhino\"}, ], })\n",
    "das.add_link( { \"type\": \"Similarity\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"vine\"}, {\"type\": \"Concept\", \"name\": \"snake\"}, ], })\n",
    "das.add_link( { \"type\": \"Similarity\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"ent\"}, {\"type\": \"Concept\", \"name\": \"human\"}, ], })\n",
    "das.add_link( { \"type\": \"Inheritance\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"human\"}, {\"type\": \"Concept\", \"name\": \"mammal\"}, ], })\n",
    "das.add_link( { \"type\": \"Inheritance\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"monkey\"}, {\"type\": \"Concept\", \"name\": \"mammal\"}, ], })\n",
    "das.add_link( { \"type\": \"Inheritance\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"chimp\"}, {\"type\": \"Concept\", \"name\": \"mammal\"}, ], })\n",
    "das.add_link( { \"type\": \"Inheritance\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"mammal\"}, {\"type\": \"Concept\", \"name\": \"animal\"}, ], })\n",
    "das.add_link( { \"type\": \"Inheritance\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"reptile\"}, {\"type\": \"Concept\", \"name\": \"animal\"}, ], })\n",
    "das.add_link( { \"type\": \"Inheritance\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"snake\"}, {\"type\": \"Concept\", \"name\": \"reptile\"}, ], })\n",
    "das.add_link( { \"type\": \"Inheritance\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"dinosaur\"}, {\"type\": \"Concept\", \"name\": \"reptile\"}, ], })\n",
    "das.add_link( { \"type\": \"Inheritance\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"triceratops\"}, {\"type\": \"Concept\", \"name\": \"dinosaur\"}, ], })\n",
    "das.add_link( { \"type\": \"Inheritance\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"earthworm\"}, {\"type\": \"Concept\", \"name\": \"animal\"}, ], })\n",
    "das.add_link( { \"type\": \"Inheritance\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"rhino\"}, {\"type\": \"Concept\", \"name\": \"mammal\"}, ], })\n",
    "das.add_link( { \"type\": \"Inheritance\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"vine\"}, {\"type\": \"Concept\", \"name\": \"plant\"}, ], })\n",
    "das.add_link( { \"type\": \"Inheritance\", \"targets\": [ {\"type\": \"Concept\", \"name\": \"ent\"}, {\"type\": \"Concept\", \"name\": \"plant\"}, ], }) ;"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e7a3fbc",
   "metadata": {},
   "source": [
    "Links are always asymetric, so symmetric relationships like \"Similarity\" are represented by adding two links. For instance:\n",
    "    \n",
    "```\n",
    "das.add_link(\n",
    "    {\n",
    "        \"type\": \"Similarity\",\n",
    "        \"targets\": [\n",
    "            {\"type\": \"Concept\", \"name\": \"human\"},\n",
    "            {\"type\": \"Concept\", \"name\": \"monkey\"},\n",
    "        ],\n",
    "    }\n",
    ")\n",
    "```\n",
    "\n",
    "and\n",
    "\n",
    "```\n",
    "das.add_link(\n",
    "    {\n",
    "        \"type\": \"Similarity\",\n",
    "        \"targets\": [\n",
    "            {\"type\": \"Concept\", \"name\": \"monkey\"},\n",
    "            {\"type\": \"Concept\", \"name\": \"human\"},\n",
    "        ],\n",
    "    }\n",
    ")\n",
    "```\n",
    "\n",
    "Considering this, we can print the atom count again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "eba2fc28",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(14, 26)\n"
     ]
    }
   ],
   "source": [
    "print(das.count_atoms())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "019ddd27",
   "metadata": {},
   "source": [
    "<a id='fetch'></a>\n",
    "### Fetching from a DAS server\n",
    "\n",
    "Instead of adding atoms by calling `add_node()` and `add_link()` directly, it's possible to fetch all or part of the contents from a DAS server using the method `fetch()`. This method doesn't create a lasting connection with the DAS server, it will just fetch the atoms once and close the connection so any subsequent changes or queries will not be propagated to the server in any way. After fetching the atoms, all queries will be made locally. It's possible to call `fetch()` multiple times fetching from the same DAS Server or from different ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "929c42bc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0, 0)\n",
      "(15, 12)\n"
     ]
    }
   ],
   "source": [
    "from hyperon_das import DistributedAtomSpace\n",
    "\n",
    "remote_das_host = \"45.63.85.59\"\n",
    "remote_das_port = 8080\n",
    "\n",
    "imported_das = DistributedAtomSpace()\n",
    "print(imported_das.count_atoms())\n",
    "\n",
    "links_to_import = {\n",
    "    'atom_type': 'link',\n",
    "    'type': 'Expression',\n",
    "    'targets': [\n",
    "        {'atom_type': 'node', 'type': 'Symbol', 'name': 'Inheritance'},\n",
    "        {'atom_type': 'variable', 'name': 'v2'},\n",
    "        {'atom_type': 'variable', 'name': 'v3'},\n",
    "    ]\n",
    "}\n",
    "\n",
    "imported_das.fetch(links_to_import, remote_das_host, remote_das_port)\n",
    "print(imported_das.count_atoms())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd1d8279",
   "metadata": {},
   "source": [
    "The first parameter of `fetch()` is a pattern to describe which atoms should be fetched. It's exactly the same pattern used to make [pattern matching](#patternmatcher)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5775788a",
   "metadata": {},
   "source": [
    "<a id='atomquery'></a>\n",
    "### Getting atoms by their properties\n",
    "\n",
    "DAS has an API to query atoms by their properties. Most of this API is based on atom handles. Handles are MD5 signatures associated with atoms. For now they are supposed to be unique ids for atoms although this is not 100% true (conflict handling is planned to be implemented in the near future). DAS provides two static methods to compute handles for nodes and links: `das.get_node_handle()` and `das.get_link_handle()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "050a3018",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "human: af12f10f9ae2002a1607ba0b47ba8407\n",
      "ent: 4e8e26e3276af8a5c2ac2cc2dc95c6d2\n",
      "Similarity link: 16f7e407087bfa0b35b13d13a1aadcae\n"
     ]
    }
   ],
   "source": [
    "human = das.get_node_handle('Concept', 'human')\n",
    "ent = das.get_node_handle('Concept', 'ent')\n",
    "\n",
    "print(\"human:\", human)\n",
    "print(\"ent:\", ent)\n",
    "\n",
    "similarity_link = das.get_link_handle('Similarity', [human, ent])\n",
    "\n",
    "print(\"Similarity link:\", similarity_link)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ec93856",
   "metadata": {},
   "source": [
    "Note that these are static methods which don't actually query the stored atomspace in order to compute those handles. Instead, they just run a MD5 hashing algorithm over the data that uniquely identifies nodes and links, i.e. node type and name in the case of nodes and link type and targets in the case of links. This means e.g. that two nodes with the same type and the same name are considered to be the exact same entity.\n",
    "\n",
    "Atom handles can be used to retrieve the actual atom document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "62c66595",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'handle': 'af12f10f9ae2002a1607ba0b47ba8407',\n",
       " 'type': 'Concept',\n",
       " 'composite_type_hash': 'd99a604c79ce3c2e76a2f43488d5d4c3',\n",
       " 'name': 'human',\n",
       " 'named_type': 'Concept'}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "das.get_atom(human)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "909812ae",
   "metadata": {},
   "source": [
    "Convenience methods can be used to retrieve atoms passing its basic properties instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f52fc680",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "human: {'handle': 'af12f10f9ae2002a1607ba0b47ba8407', 'type': 'Concept', 'composite_type_hash': 'd99a604c79ce3c2e76a2f43488d5d4c3', 'name': 'human', 'named_type': 'Concept'}\n",
      "\n",
      "Similarity link: {'handle': '16f7e407087bfa0b35b13d13a1aadcae', 'type': 'Similarity', 'composite_type_hash': 'ed73ea081d170e1d89fc950820ce1cee', 'is_toplevel': True, 'composite_type': ['a9dea78180588431ec64d6bc4872fdbc', 'd99a604c79ce3c2e76a2f43488d5d4c3', 'd99a604c79ce3c2e76a2f43488d5d4c3'], 'named_type': 'Similarity', 'named_type_hash': 'a9dea78180588431ec64d6bc4872fdbc', 'targets': ['af12f10f9ae2002a1607ba0b47ba8407', '4e8e26e3276af8a5c2ac2cc2dc95c6d2']}\n"
     ]
    }
   ],
   "source": [
    "print(\"human:\", das.get_node('Concept', 'human'))\n",
    "print(\"\\nSimilarity link:\", das.get_link('Similarity', [human, ent]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afc014d9",
   "metadata": {},
   "source": [
    "It's possible to get all links pointing to a specific atom."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1161da00",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity ['d03e59654221c1e8fcda404fd5c8d6cb', '99d18c702e813b07260baf577c60c455']\n",
      "Similarity ['99d18c702e813b07260baf577c60c455', 'd03e59654221c1e8fcda404fd5c8d6cb']\n",
      "Inheritance ['99d18c702e813b07260baf577c60c455', 'bdfe4e7a431f73386f37c6448afe5840']\n"
     ]
    }
   ],
   "source": [
    "# All links pointing from/to 'rhino'\n",
    "\n",
    "rhino = das.get_node_handle('Concept', 'rhino')\n",
    "links = das.get_incoming_links(rhino)\n",
    "for link in links:\n",
    "    print(link['type'], link['targets'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "433e1a9b",
   "metadata": {},
   "source": [
    "Links can also be retrieved by other properties or partial definition of its main properties (type and targets). The method `get_links()` can be used passing different combinations of parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a69064a6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inheritance ['5b34c54bee150c04f9fa584b899dc030', 'bdfe4e7a431f73386f37c6448afe5840']\n",
      "Inheritance ['b94941d8cd1c0ee4ad3dd3dcab52b964', '80aff30094874e75028033a38ce677bb']\n",
      "Inheritance ['bb34ce95f161a6b37ff54b3d4c817857', '0a32b476852eeb954979b87f5f6cb7af']\n",
      "Inheritance ['c1db9b517073e51eb7ef6fed608ec204', 'b99ae727c787f1b13b452fd4c9ce1b9a']\n",
      "Inheritance ['bdfe4e7a431f73386f37c6448afe5840', '0a32b476852eeb954979b87f5f6cb7af']\n",
      "Inheritance ['1cdffc6b0b89ff41d68bec237481d1e1', 'bdfe4e7a431f73386f37c6448afe5840']\n",
      "Inheritance ['af12f10f9ae2002a1607ba0b47ba8407', 'bdfe4e7a431f73386f37c6448afe5840']\n",
      "Inheritance ['b99ae727c787f1b13b452fd4c9ce1b9a', '0a32b476852eeb954979b87f5f6cb7af']\n",
      "Inheritance ['4e8e26e3276af8a5c2ac2cc2dc95c6d2', '80aff30094874e75028033a38ce677bb']\n",
      "Inheritance ['d03e59654221c1e8fcda404fd5c8d6cb', '08126b066d32ee37743e255a2558cccd']\n",
      "Inheritance ['99d18c702e813b07260baf577c60c455', 'bdfe4e7a431f73386f37c6448afe5840']\n",
      "Inheritance ['08126b066d32ee37743e255a2558cccd', 'b99ae727c787f1b13b452fd4c9ce1b9a']\n"
     ]
    }
   ],
   "source": [
    "# All inheritance links\n",
    "\n",
    "links = das.get_links(link_type='Inheritance')\n",
    "for link in links:\n",
    "    print(link['type'], link['targets'])                      "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "836d64d8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inheritance ['5b34c54bee150c04f9fa584b899dc030', 'bdfe4e7a431f73386f37c6448afe5840']\n",
      "Inheritance ['b94941d8cd1c0ee4ad3dd3dcab52b964', '80aff30094874e75028033a38ce677bb']\n",
      "Inheritance ['bb34ce95f161a6b37ff54b3d4c817857', '0a32b476852eeb954979b87f5f6cb7af']\n",
      "Inheritance ['c1db9b517073e51eb7ef6fed608ec204', 'b99ae727c787f1b13b452fd4c9ce1b9a']\n",
      "Inheritance ['bdfe4e7a431f73386f37c6448afe5840', '0a32b476852eeb954979b87f5f6cb7af']\n",
      "Inheritance ['1cdffc6b0b89ff41d68bec237481d1e1', 'bdfe4e7a431f73386f37c6448afe5840']\n",
      "Inheritance ['af12f10f9ae2002a1607ba0b47ba8407', 'bdfe4e7a431f73386f37c6448afe5840']\n",
      "Inheritance ['b99ae727c787f1b13b452fd4c9ce1b9a', '0a32b476852eeb954979b87f5f6cb7af']\n",
      "Inheritance ['4e8e26e3276af8a5c2ac2cc2dc95c6d2', '80aff30094874e75028033a38ce677bb']\n",
      "Inheritance ['d03e59654221c1e8fcda404fd5c8d6cb', '08126b066d32ee37743e255a2558cccd']\n",
      "Inheritance ['99d18c702e813b07260baf577c60c455', 'bdfe4e7a431f73386f37c6448afe5840']\n",
      "Inheritance ['08126b066d32ee37743e255a2558cccd', 'b99ae727c787f1b13b452fd4c9ce1b9a']\n"
     ]
    }
   ],
   "source": [
    "# Inheritance links between two Concept nodes\n",
    "\n",
    "links = das.get_links(link_type='Inheritance', target_types=['Concept', 'Concept'])\n",
    "for link in links:\n",
    "    print(link['type'], link['targets'])   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "30e8125c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity ['c1db9b517073e51eb7ef6fed608ec204', 'b94941d8cd1c0ee4ad3dd3dcab52b964']\n",
      "Similarity ['c1db9b517073e51eb7ef6fed608ec204', 'bb34ce95f161a6b37ff54b3d4c817857']\n"
     ]
    }
   ],
   "source": [
    "# Similarity links where 'snake' is the first target\n",
    "\n",
    "snake = das.get_node_handle('Concept', 'snake')\n",
    "links = das.get_links(link_type='Similarity', link_targets=[snake, '*'])\n",
    "for link in links:\n",
    "    print(link['type'], link['targets']) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c3fd51b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity ['c1db9b517073e51eb7ef6fed608ec204', 'b94941d8cd1c0ee4ad3dd3dcab52b964']\n",
      "Inheritance ['c1db9b517073e51eb7ef6fed608ec204', 'b99ae727c787f1b13b452fd4c9ce1b9a']\n",
      "Similarity ['c1db9b517073e51eb7ef6fed608ec204', 'bb34ce95f161a6b37ff54b3d4c817857']\n"
     ]
    }
   ],
   "source": [
    "# Any links where 'snake' is the first target\n",
    "\n",
    "snake = das.get_node_handle('Concept', 'snake')\n",
    "links = das.get_links(link_type='*', link_targets=[snake, '*'])\n",
    "for link in links:\n",
    "    print(link['type'], link['targets']) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b6e1aed",
   "metadata": {},
   "source": [
    "<a id='traversing'></a>\n",
    "### Traversing the hypergraph\n",
    "\n",
    "It's possible to traverse the hypergraph using a `TraverseEngine` which is like a cursor that can be moved through nodes and links. First, let's initiate a `TraverseEngine` pointing to \"human\". In order to do this, we need to call `get_traversal_cursor()` passing the handle of the atom to be used as the starting point for the traversing. This atom can be either a link or a node. We'll use the method `das.get_node_handle()` to get the handle of the Concept \"human\" and start on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "b5203c32",
   "metadata": {},
   "outputs": [],
   "source": [
    "cursor = das.get_traversal_cursor(das.get_node_handle('Concept', 'human'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "039c8d37",
   "metadata": {},
   "source": [
    "Once we have a cursor we can get the whole document of the atom pointed by it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d4d7b6a6",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'handle': 'af12f10f9ae2002a1607ba0b47ba8407',\n",
       " 'type': 'Concept',\n",
       " 'composite_type_hash': 'd99a604c79ce3c2e76a2f43488d5d4c3',\n",
       " 'name': 'human',\n",
       " 'named_type': 'Concept'}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cursor.get()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e4d7d25",
   "metadata": {},
   "source": [
    "We can also see all links that make reference to cursor. Optional parameters can be used to filter which links should be considered. Here are some examples. We're printing only link type and targets to make the output cleaner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "357e912e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All links: [('Similarity', ['af12f10f9ae2002a1607ba0b47ba8407', '4e8e26e3276af8a5c2ac2cc2dc95c6d2']), ('Inheritance', ['af12f10f9ae2002a1607ba0b47ba8407', 'bdfe4e7a431f73386f37c6448afe5840']), ('Similarity', ['af12f10f9ae2002a1607ba0b47ba8407', '5b34c54bee150c04f9fa584b899dc030']), ('Similarity', ['1cdffc6b0b89ff41d68bec237481d1e1', 'af12f10f9ae2002a1607ba0b47ba8407']), ('Similarity', ['4e8e26e3276af8a5c2ac2cc2dc95c6d2', 'af12f10f9ae2002a1607ba0b47ba8407']), ('Similarity', ['af12f10f9ae2002a1607ba0b47ba8407', '1cdffc6b0b89ff41d68bec237481d1e1']), ('Similarity', ['5b34c54bee150c04f9fa584b899dc030', 'af12f10f9ae2002a1607ba0b47ba8407'])]\n",
      "\n",
      "Inheritance links: [('Inheritance', ['af12f10f9ae2002a1607ba0b47ba8407', 'bdfe4e7a431f73386f37c6448afe5840'])]\n",
      "\n",
      "'human' is first link target: [('Similarity', ['af12f10f9ae2002a1607ba0b47ba8407', '4e8e26e3276af8a5c2ac2cc2dc95c6d2']), ('Inheritance', ['af12f10f9ae2002a1607ba0b47ba8407', 'bdfe4e7a431f73386f37c6448afe5840']), ('Similarity', ['af12f10f9ae2002a1607ba0b47ba8407', '5b34c54bee150c04f9fa584b899dc030']), ('Similarity', ['af12f10f9ae2002a1607ba0b47ba8407', '1cdffc6b0b89ff41d68bec237481d1e1'])]\n"
     ]
    }
   ],
   "source": [
    "# All links pointing from/to cursor\n",
    "print(\"All links:\", [(d['type'], d['targets']) for d in cursor.get_links()])\n",
    "\n",
    "# Only Inheritance links\n",
    "print(\"\\nInheritance links:\", [(d['type'], d['targets']) for d in cursor.get_links(link_type='Inheritance')])\n",
    "\n",
    "# Links whose first target is our cursor\n",
    "print(\"\\n'human' is first link target:\", [(d['type'], d['targets']) for d in cursor.get_links(cursor_position=0)])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b535aa8",
   "metadata": {},
   "source": [
    "There are other possibilities for filtering such as custom filter methods, target types, etc. They're explained in the [DAS API](https://singnet.github.io/das-query-engine/api/das/).\n",
    "\n",
    "There are also convenience methods to get the cursor's \"neighbors\", which are the other atoms pointed by the links attached to the cursor. Let's investigate the neighbors of \"human\". Again, we can use the same filters to select which links and targets to consider in order to get the neighbors of the cursor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "63e2dbdd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All neighbors: [('Concept', 'ent'), ('Concept', 'mammal'), ('Concept', 'chimp'), ('Concept', 'monkey')]\n",
      "\n",
      "Inheritance relations: [('Concept', 'mammal')]\n",
      "\n",
      "Similar to 'human': [('Concept', 'ent'), ('Concept', 'chimp'), ('Concept', 'monkey')]\n"
     ]
    }
   ],
   "source": [
    "# All \"human\" neighbors\n",
    "print(\"All neighbors:\", [(d['type'], d['name']) for d in cursor.get_neighbors()])\n",
    "\n",
    "# Only neighbors linked through Inheritance links\n",
    "print(\"\\nInheritance relations:\", [(d['type'], d['name']) for d in cursor.get_neighbors(link_type='Inheritance')])\n",
    "\n",
    "# Only neighbors that are similar to \"human\" (i.e. they share a Similarity link)\n",
    "print(\"\\nSimilar to 'human':\", [(d['type'], d['name']) for d in cursor.get_neighbors(link_type='Similarity', cursor_position=0)])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90a9b57b",
   "metadata": {},
   "source": [
    "<a id='cache'></a>\n",
    "`get_links()` and `get_neighbors()` use the [DAS Cache system](https://github.com/singnet/das/blob/master/docs/das-overview.md) to sort the atoms before they are returned to the caller. In addition to this, these methods return an iterator rather than an actual list of atoms and this iterator is controlled by the cache system as well. The idea here is that atoms may have a large number of links (and consequently neighbors) attached to it so the AI/ML agent may not be interested in iterating on all of them. Atoms are presented in such a way that high importance atoms tend to be presented first while low importance atoms tend to be presented later.\n",
    "\n",
    "We can move the cursor by following its links. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "e3ee2ca0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'handle': '4e8e26e3276af8a5c2ac2cc2dc95c6d2',\n",
       " 'type': 'Concept',\n",
       " 'composite_type_hash': 'd99a604c79ce3c2e76a2f43488d5d4c3',\n",
       " 'name': 'ent',\n",
       " 'named_type': 'Concept'}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cursor = das.get_traversal_cursor(das.get_node_handle('Concept', 'human'))\n",
    "cursor.follow_link()\n",
    "cursor.get()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ae21850",
   "metadata": {},
   "source": [
    "`follow_link()` just gets the first link returned by `get_links()` in order to follow it and select a target. The same filters described above can be used here to constraint the links/targets that will be considered. For instance we could use the following code to get the most abstract concept (considering our Inheritance links) starting from \"human\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "5569f856",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "human\n",
      "mammal\n",
      "animal\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'handle': '0a32b476852eeb954979b87f5f6cb7af',\n",
       " 'type': 'Concept',\n",
       " 'composite_type_hash': 'd99a604c79ce3c2e76a2f43488d5d4c3',\n",
       " 'name': 'animal',\n",
       " 'named_type': 'Concept'}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cursor = das.get_traversal_cursor(das.get_node_handle('Concept', 'human'))\n",
    "base = cursor.get()['name']\n",
    "while True:\n",
    "    print(base)\n",
    "    cursor.follow_link(link_type='Inheritance', cursor_position=0)\n",
    "    if cursor.get()['name'] == base:\n",
    "        break\n",
    "    base = cursor.get()['name']\n",
    "cursor.get()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "524c947d",
   "metadata": {},
   "source": [
    "<a id='patternmatcher'></a>\n",
    "### Pattern Matcher Queries\n",
    "\n",
    "DAS can answer pattern matching queries. These are queries where the caller specifies a _pattern_ i.e. a boolean expression of subgraphs with nodes, links and wildcards and the engine finds every subgraph in the knowledge base that satisfies the passed expression. Patterns are a list of Python dicts describing a subgraph with wildcards.\n",
    "\n",
    "The method `query()` expects a pattern and outputs a list of `QueryAnswer`. Each element in such a list has the variable assignment that satisfies the pattern and the subgraph which is the pattern itself rewritten using the given assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "c629e479",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('v1', 'b94941d8cd1c0ee4ad3dd3dcab52b964')]\n",
      "v1: Concept vine\n",
      "{'handle': 'e4685d56969398253b6f77efd21dc347', 'type': 'Inheritance', 'composite_type_hash': '41c082428b28d7e9ea96160f7fd614ad', 'is_toplevel': True, 'composite_type': ['e40489cd1e7102e35469c937e05c8bba', 'd99a604c79ce3c2e76a2f43488d5d4c3', 'd99a604c79ce3c2e76a2f43488d5d4c3'], 'named_type': 'Inheritance', 'named_type_hash': 'e40489cd1e7102e35469c937e05c8bba', 'targets': [{'handle': 'b94941d8cd1c0ee4ad3dd3dcab52b964', 'type': 'Concept', 'composite_type_hash': 'd99a604c79ce3c2e76a2f43488d5d4c3', 'name': 'vine', 'named_type': 'Concept'}, {'handle': '80aff30094874e75028033a38ce677bb', 'type': 'Concept', 'composite_type_hash': 'd99a604c79ce3c2e76a2f43488d5d4c3', 'name': 'plant', 'named_type': 'Concept'}]}\n",
      "\n",
      "[('v1', '4e8e26e3276af8a5c2ac2cc2dc95c6d2')]\n",
      "v1: Concept ent\n",
      "{'handle': 'ee1c03e6d1f104ccd811cfbba018451a', 'type': 'Inheritance', 'composite_type_hash': '41c082428b28d7e9ea96160f7fd614ad', 'is_toplevel': True, 'composite_type': ['e40489cd1e7102e35469c937e05c8bba', 'd99a604c79ce3c2e76a2f43488d5d4c3', 'd99a604c79ce3c2e76a2f43488d5d4c3'], 'named_type': 'Inheritance', 'named_type_hash': 'e40489cd1e7102e35469c937e05c8bba', 'targets': [{'handle': '4e8e26e3276af8a5c2ac2cc2dc95c6d2', 'type': 'Concept', 'composite_type_hash': 'd99a604c79ce3c2e76a2f43488d5d4c3', 'name': 'ent', 'named_type': 'Concept'}, {'handle': '80aff30094874e75028033a38ce677bb', 'type': 'Concept', 'composite_type_hash': 'd99a604c79ce3c2e76a2f43488d5d4c3', 'name': 'plant', 'named_type': 'Concept'}]}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# This is a pattern like:\n",
    "#\n",
    "# Inheritance\n",
    "#     v1\n",
    "#     plant\n",
    "#\n",
    "# The expected answer is all Inheritance links whose second target == 'plant'\n",
    "#\n",
    "query = {\n",
    "    'atom_type': 'link',\n",
    "    'type': 'Inheritance',\n",
    "    'targets': [\n",
    "        {'atom_type': 'variable', 'name': 'v1'},\n",
    "        {'atom_type': 'node', 'type': 'Concept', 'name': 'plant'},\n",
    "    ]\n",
    "}\n",
    "\n",
    "for query_answer in das.query(query):\n",
    "    print(query_answer.assignment)\n",
    "    atom_matching_v1 = das.get_atom(query_answer.assignment.mapping['v1'])\n",
    "    print(\"v1:\", atom_matching_v1['type'], atom_matching_v1['name'])\n",
    "    rewrited_query = query_answer.subgraph\n",
    "    print(rewrited_query)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "ae33b933",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('v1', '99d18c702e813b07260baf577c60c455'), ('v2', 'd03e59654221c1e8fcda404fd5c8d6cb')]\n",
      "v1: Concept rhino\n",
      "v2: Concept triceratops\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# This is a pattern like:\n",
    "#\n",
    "# AND\n",
    "#     Inheritance\n",
    "#         v1\n",
    "#         mammal\n",
    "#     Inheritance\n",
    "#         v2\n",
    "#         dinosaur\n",
    "#     Similarity\n",
    "#         v1\n",
    "#         v2\n",
    "#\n",
    "# The expected answer is all pair of animals such that \n",
    "# one inherits from mammal, the other inherits from dinosaur \n",
    "# and they have a Similarity link between them.\n",
    "#\n",
    "exp1 = {\n",
    "    'atom_type': 'link',\n",
    "    'type': 'Inheritance',\n",
    "    'targets': [\n",
    "        {'atom_type': 'variable', 'name': 'v1'},\n",
    "        {'atom_type': 'node', 'type': 'Concept', 'name': 'mammal'},\n",
    "    ]\n",
    "}\n",
    "exp2 = {\n",
    "    'atom_type': 'link',\n",
    "    'type': 'Inheritance',\n",
    "    'targets': [\n",
    "        {'atom_type': 'variable', 'name': 'v2'},\n",
    "        {'atom_type': 'node', 'type': 'Concept', 'name': 'dinosaur'},\n",
    "    ]\n",
    "}\n",
    "exp3 = {\n",
    "    'atom_type': 'link',\n",
    "    'type': 'Similarity',\n",
    "    'targets': [\n",
    "        {'atom_type': 'variable', 'name': 'v1'},\n",
    "        {'atom_type': 'variable', 'name': 'v2'},\n",
    "    ]\n",
    "}\n",
    "query = [exp1, exp2, exp3] # a list of expressions mean an AND of them\n",
    "\n",
    "for query_answer in das.query(query):\n",
    "    print(query_answer.assignment)\n",
    "    atom_matching_v1 = das.get_atom(query_answer.assignment.mapping['v1'])\n",
    "    atom_matching_v2 = das.get_atom(query_answer.assignment.mapping['v2'])\n",
    "    print(\"v1:\", atom_matching_v1['type'], atom_matching_v1['name'])\n",
    "    print(\"v2:\", atom_matching_v2['type'], atom_matching_v2['name'])\n",
    "    #rewrited_query = query_answer.subgraph\n",
    "    #print(rewrited_query)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "92930b26",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "v1: Concept monkey\n",
      "v2: Concept chimp\n",
      "v3: Concept human\n",
      "\n",
      "v1: Concept human\n",
      "v2: Concept monkey\n",
      "v3: Concept chimp\n",
      "\n",
      "v1: Concept chimp\n",
      "v2: Concept monkey\n",
      "v3: Concept human\n",
      "\n",
      "v1: Concept monkey\n",
      "v2: Concept human\n",
      "v3: Concept chimp\n",
      "\n",
      "v1: Concept human\n",
      "v2: Concept chimp\n",
      "v3: Concept monkey\n",
      "\n",
      "v1: Concept chimp\n",
      "v2: Concept human\n",
      "v3: Concept monkey\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# This is a pattern like:\n",
    "#\n",
    "# AND\n",
    "#     Similarity\n",
    "#         v1\n",
    "#         v2\n",
    "#     Similarity\n",
    "#         v2\n",
    "#         v3\n",
    "#     Similarity\n",
    "#         v3\n",
    "#         v1\n",
    "#\n",
    "# The expected answer is all triplet of animals such that \n",
    "# all of them have a Similarity link with the other two.\n",
    "#\n",
    "exp1 = {\n",
    "    'atom_type': 'link',\n",
    "    'type': 'Similarity',\n",
    "    'targets': [\n",
    "        {'atom_type': 'variable', 'name': 'v1'},\n",
    "        {'atom_type': 'variable', 'name': 'v2'},\n",
    "    ]\n",
    "}\n",
    "exp2 = {\n",
    "    'atom_type': 'link',\n",
    "    'type': 'Similarity',\n",
    "    'targets': [\n",
    "        {'atom_type': 'variable', 'name': 'v2'},\n",
    "        {'atom_type': 'variable', 'name': 'v3'},\n",
    "    ]\n",
    "}\n",
    "exp3 = {\n",
    "    'atom_type': 'link',\n",
    "    'type': 'Similarity',\n",
    "    'targets': [\n",
    "        {'atom_type': 'variable', 'name': 'v3'},\n",
    "        {'atom_type': 'variable', 'name': 'v1'},\n",
    "    ]\n",
    "}\n",
    "query = [exp1, exp2, exp3] # a list of expressions mean an AND of them\n",
    "\n",
    "for query_answer in das.query(query):\n",
    "    atom_matching_v1 = das.get_atom(query_answer.assignment.mapping['v1'])\n",
    "    atom_matching_v2 = das.get_atom(query_answer.assignment.mapping['v2'])\n",
    "    atom_matching_v3 = das.get_atom(query_answer.assignment.mapping['v3'])\n",
    "    print(\"v1:\", atom_matching_v1['type'], atom_matching_v1['name'])\n",
    "    print(\"v2:\", atom_matching_v2['type'], atom_matching_v2['name'])\n",
    "    print(\"v3:\", atom_matching_v3['type'], atom_matching_v3['name'])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9560897",
   "metadata": {},
   "source": [
    "<a id='remotedas'></a>\n",
    "## Connecting to a remote DAS"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c34885b",
   "metadata": {},
   "source": [
    "When a DAS is instantiated with a remote query engine, it will connect to a DAS Server previously populated with a knowledge base. Atoms in the remote DAS Server become available for fetching, querying and modification.\n",
    "\n",
    "In addition to the remote DAS, an internal local DAS is also kept locally. Some of the methods in the API will look for atoms first in this local DAS before going to the remote one. Other methods can be configured to search only in one of them (remote or local) or in both. We'll explain this behavior on a case by case basis.\n",
    "\n",
    "In our example, we'll connect to a DAS Server pre-loaded with the following MeTTa expressions:\n",
    "\n",
    "```\n",
    "(: Similarity Type)\n",
    "(: Concept Type)\n",
    "(: Inheritance Type)\n",
    "(: \"human\" Concept)\n",
    "(: \"monkey\" Concept)\n",
    "(: \"chimp\" Concept)\n",
    "(: \"snake\" Concept)\n",
    "(: \"earthworm\" Concept)\n",
    "(: \"rhino\" Concept)\n",
    "(: \"triceratops\" Concept)\n",
    "(: \"vine\" Concept)\n",
    "(: \"ent\" Concept)\n",
    "(: \"mammal\" Concept)\n",
    "(: \"animal\" Concept)\n",
    "(: \"reptile\" Concept)\n",
    "(: \"dinosaur\" Concept)\n",
    "(: \"plant\" Concept)\n",
    "(Similarity \"human\" \"monkey\")\n",
    "(Similarity \"human\" \"chimp\")\n",
    "(Similarity \"chimp\" \"monkey\")\n",
    "(Similarity \"snake\" \"earthworm\")\n",
    "(Similarity \"rhino\" \"triceratops\")\n",
    "(Similarity \"snake\" \"vine\")\n",
    "(Similarity \"human\" \"ent\")\n",
    "(Inheritance \"human\" \"mammal\")\n",
    "(Inheritance \"monkey\" \"mammal\")\n",
    "(Inheritance \"chimp\" \"mammal\")\n",
    "(Inheritance \"mammal\" \"animal\")\n",
    "(Inheritance \"reptile\" \"animal\")\n",
    "(Inheritance \"snake\" \"reptile\")\n",
    "(Inheritance \"dinosaur\" \"reptile\")\n",
    "(Inheritance \"triceratops\" \"dinosaur\")\n",
    "(Inheritance \"earthworm\" \"animal\")\n",
    "(Inheritance \"rhino\" \"mammal\")\n",
    "(Inheritance \"vine\" \"plant\")\n",
    "(Inheritance \"ent\" \"plant\")\n",
    "(Similarity \"monkey\" \"human\")\n",
    "(Similarity \"chimp\" \"human\")\n",
    "(Similarity \"monkey\" \"chimp\")\n",
    "(Similarity \"earthworm\" \"snake\")\n",
    "(Similarity \"triceratops\" \"rhino\")\n",
    "(Similarity \"vine\" \"snake\")\n",
    "(Similarity \"ent\" \"human\")\n",
    "```\n",
    "\n",
    "Semantically, this is the same knowledge base we used as an example for a local DAS above. However, the mapping to nodes and links is slightly different as described in the [DAS MeTTa Parser](https://github.com/singnet/das-metta-parser) documentation. For instance, each expression, like:\n",
    "\n",
    "```\n",
    "(Similarity \"ent\" \"human\")\n",
    "```\n",
    "\n",
    "is mapped to 4 atoms. 3 nodes and 1 link as follows.\n",
    "\n",
    "```\n",
    "{\n",
    "    'type': 'Expression',\n",
    "    'targets': [\n",
    "        {'type': 'Symbol', 'name', 'Similarity'},\n",
    "        {'type': 'Symbol', 'name', '\"ent\"'},\n",
    "        {'type': 'Symbol', 'name', '\"human\"'}\n",
    "    ]\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "0ba8c216",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connected to DAS Server at 45.63.85.59:8080\n",
      "(nodes, links) = (23, 60)\n"
     ]
    }
   ],
   "source": [
    "from hyperon_das import DistributedAtomSpace\n",
    "\n",
    "host = '45.63.85.59'\n",
    "port = '8080'\n",
    "\n",
    "remote_das = DistributedAtomSpace(query_engine='remote', host=host, port=port)\n",
    "print(f\"Connected to DAS Server at {host}:{port}\")\n",
    "\n",
    "print(\"(nodes, links) =\", remote_das.count_atoms())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "536f5aa4",
   "metadata": {},
   "source": [
    "Atoms can be retrieved by their properties using `get_atom()`, `get_node()`, `get_link()`, `get_incoming_links()` and `get_links()` in the same way described [here](#atomquery) for local DAS. The only difference is that the local DAS will be searched first for `get_atom()`, `get_node()`, `get_link()` before going to the remote DAS when the atom is not found locally. `get_incoming_links()` and `get_links()` will search in both, local and remote DAS, and return an iterator to the results. As we explain [here](#cache), these iterators use the cache system to sort the results and determine how atoms are fetched from the remote DAS.\n",
    "\n",
    "`add_node()` and `add_link()` will add atoms only in the local DAS. If you add an atom that already exists in the remote DAS, the local copy is always returned by the methods above. To propagate changes to the remote DAS one needs to call `commit()`. We'll not provide examples of changes in the remote DAS here because we're using a single DAS Server to serve tests with this animals KB so if you commit changes to it everyone will be affected. So please don't use this notebook to commit changes to our test server.\n",
    "\n",
    "`fetch()` also works in the same way (described [here](#fetch)) for a remote DAS. The only difference is that now the caller can omit the parameters for `host` and `port` which are defaulted to the connected remote DAS Server. Fetching from a different DAS Server is still possible by setting the proper values for `host` and `port`.\n",
    "\n",
    "If you execute the cells below you'll notice a delay between each call. This is because the cache system is not in place yet so each call is issuing an actual query to the remote DAS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "5e403f5c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'handle': '665509d366ac3c2821b3b6b266f996bd',\n",
       " 'type': 'Symbol',\n",
       " 'composite_type_hash': '02c86eb2792f3262c21d030a87e19793',\n",
       " 'name': '\"earthworm\"',\n",
       " 'named_type': 'Symbol',\n",
       " 'is_literal': True}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute the handle and get the actual document for \"symbol\"\n",
    "symbol = '\"earthworm\"'\n",
    "symbol_handle = remote_das.get_node_handle('Symbol', symbol)\n",
    "symbol_document = remote_das.get_atom(symbol_handle)\n",
    "symbol_document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "4af804cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(: \"earthworm\" Concept)\n",
      "(Inheritance \"earthworm\" \"animal\")\n",
      "(Similarity \"earthworm\" \"snake\")\n"
     ]
    }
   ],
   "source": [
    "# Get expressions like (* base_symbol *)\n",
    "iterator = remote_das.get_links(link_type='Expression', link_targets=['*', symbol_handle, '*'])\n",
    "for link in iterator:\n",
    "    atom1 = remote_das.get_atom(link['targets'][0])\n",
    "    atom2 = remote_das.get_atom(link['targets'][2])\n",
    "    print(f\"({atom1['name']} {symbol} {atom2['name']})\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "6ea0e2d0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'handle': '665509d366ac3c2821b3b6b266f996bd',\n",
       " 'type': 'Symbol',\n",
       " 'composite_type_hash': '02c86eb2792f3262c21d030a87e19793',\n",
       " 'name': '\"earthworm\"',\n",
       " 'named_type': 'Symbol',\n",
       " 'truth_value': (0.1, 0.9)}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Re-adding an existing atom with a custom field\n",
    "remote_das.add_node(\n",
    "    {\n",
    "        'type': 'Symbol',\n",
    "        'name': symbol,\n",
    "        'truth_value': tuple([0.1, 0.9])\n",
    "    }\n",
    ")\n",
    "remote_das.get_node('Symbol', symbol)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "80d2ad9e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(Pos \"earthworm\" noun)\n",
      "(: \"earthworm\" Concept)\n",
      "(Inheritance \"earthworm\" \"animal\")\n",
      "(Similarity \"earthworm\" \"snake\")\n"
     ]
    }
   ],
   "source": [
    "# Add (to the local DAS only) a new expression mentioning the base_symbol\n",
    "remote_das.add_link(\n",
    "    { \n",
    "        'type': 'Expression', \n",
    "        'targets': [ \n",
    "            {'type': 'Symbol', 'name': 'Pos'}, \n",
    "            {'type': 'Symbol', 'name': symbol},\n",
    "            {'type': 'Symbol', 'name': 'noun'}\n",
    "        ]\n",
    "    }\n",
    ")\n",
    "# Get expressions like (* base_symbol *) again\n",
    "iterator = remote_das.get_links(link_type='Expression', link_targets=['*', symbol_handle, '*'])\n",
    "for link in iterator:\n",
    "    atom1 = remote_das.get_atom(link['targets'][0])\n",
    "    atom2 = remote_das.get_atom(link['targets'][2])\n",
    "    print(f\"({atom1['name']} {symbol} {atom2['name']})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bcd86be",
   "metadata": {},
   "source": [
    "The methods for traversing the hypergraph work basically in the same way as for the local DAS (this is described [here](#traversing)). Because of the way MeTTa expressions are mapped to nodes/links with only one type of node and one type of link, traversing is less intuitive from a human perspective but it still makes sense to implement algorithms. Local and remote DAS are considered by the `TraverseEngine` and the whole logic of this component is subject to the cache management rules, i.e., the cache will try to pre-fetch atoms and present query answers prioritizing more relevant atoms as the caller navigates through the atomspace hypergraph."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16c8a2ff",
   "metadata": {},
   "source": [
    "<a id='remotequery'></a>\n",
    "### Querying a remote DAS\n",
    "\n",
    "The Pattern Matcher in a remote DAS works basically in the same way as in a local DAS (this is described [here](#patternmatcher)). The main difference is the optional parameter `query_scope` which can be used to define the scope of the query as `local_only`, `remote_only` or `local_and_remote` (its default value is `remote_only`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "9015f009",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Remote only\n",
      "(Inheritance \"earthworm\" \"animal\")\n",
      "(: \"earthworm\" Concept)\n",
      "(Similarity \"earthworm\" \"snake\")\n",
      "\n",
      "Local only\n",
      "(Pos \"earthworm\" noun)\n"
     ]
    }
   ],
   "source": [
    "query = {\n",
    "    'atom_type': 'link',\n",
    "    'type': 'Expression',\n",
    "    'targets': [\n",
    "        {'atom_type': 'variable', 'name': 'v1'},\n",
    "        {'atom_type': 'node', 'type': 'Symbol', 'name': symbol},\n",
    "        {'atom_type': 'variable', 'name': 'v2'}\n",
    "    ]\n",
    "}\n",
    "\n",
    "# The default is to query remote_only\n",
    "results = remote_das.query(query)\n",
    "print(\"Remote only\")\n",
    "for query_answer in results:\n",
    "    v1_atom = query_answer[1]['targets'][0]\n",
    "    v2_atom = query_answer[1]['targets'][2]\n",
    "    print(f\"({v1_atom['name']} {symbol} {v2_atom['name']})\")\n",
    "\n",
    "results = remote_das.query(query, {'query_scope': 'local_only'})\n",
    "print()\n",
    "print(\"Local only\")\n",
    "for query_answer in results:\n",
    "    v1_atom = query_answer.subgraph['targets'][0]\n",
    "    v2_atom = query_answer.subgraph['targets'][2]\n",
    "    print(f\"({v1_atom['name']} {symbol} {v2_atom['name']})\")\n",
    "\n",
    "# local_and_remote is not implemented yet\n",
    "#results = remote_das.query(query, {'query_scope': 'local_and_remote'})\n",
    "#print(\"Remote + Local\")\n",
    "#for query_answer in results:\n",
    "#    v1_atom = query_answer[1]['targets'][0]\n",
    "#    v2_atom = query_answer[1]['targets'][2]\n",
    "#    print(f\"({v1_atom['name']} {symbol} {v2_atom['name']})\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20eed47a",
   "metadata": {},
   "source": [
    "<a id='customindex'></a>\n",
    "### Custom Indexes\n",
    "\n",
    "Remote DAS allow creation of custom indexes based on custom fields in nodes or links. These indexes can be used to make subsequent custom queries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "e296c25f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Symbol \"human\"\n"
     ]
    }
   ],
   "source": [
    "symbol_name_index = remote_das.create_field_index('node', 'name', type='Symbol')\n",
    "results = remote_das.custom_query(symbol_name_index, name='\"human\"')\n",
    "for atom in results:\n",
    "    print(atom['type'], atom['name'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd4deef0",
   "metadata": {},
   "source": [
    "In this example, we're creating an index for the field `name` in nodes. `name` is supposed to be defined in every node of the knowledge base. To create an index on a field which is defined only for a certain type of node, an extra `type` parameter should be passed to define which type of nodes should enter in the index: e.g. `remote_das.create_field_index('node', 'lemma', type='Word')` would create an index for the field `lemma` on all nodes of type `Word`. This type of index works only for string or number (integer or floating point) fields.\n",
    "Indexes for links can be created likewise."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "586a4078",
   "metadata": {},
   "source": [
    "<a id='dasserver'></a>\n",
    "## Starting a DAS Server\n",
    "\n",
    "A DAS Server can be set up using the [DAS Toolbox](https://github.com/singnet/das-toolbox) following these steps:\n",
    "\n",
    "1. Setup environment variables\n",
    "1. Start DB servers\n",
    "1. Load MeTTa knowledge base\n",
    "1. Start FaaS gateway\n",
    "\n",
    "First, you need to install the latest version of `das-cli` in your environment. Follow the instructions in the [toolbox repo](https://github.com/singnet/das-toolbox) to make this.\n",
    "\n",
    "Then we'll start by setting up the environment.\n",
    "\n",
    "<span style=\"color:red\">*THE COMMANDS BELOW WILL CREATE FILES IN YOUR FILESYSTEM*</span>.\n",
    "\n",
    "Run the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56940cee",
   "metadata": {},
   "outputs": [],
   "source": [
    "!das-cli config list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "417937c8",
   "metadata": {},
   "source": [
    "If it outputs something like this:\n",
    "\n",
    "```\n",
    "+----------+----------------+-----------------------+\n",
    "| Service  | Name           | Value                 |\n",
    "+----------+----------------+-----------------------+\n",
    "| redis    | port           | 29000                 |\n",
    "| redis    | container_name | das-cli-redis-29000   |\n",
    "| mongodb  | port           | 28000                 |\n",
    "| mongodb  | container_name | das-cli-mongodb-28000 |\n",
    "| mongodb  | username       | dbadmin               |\n",
    "| mongodb  | password       | dassecret             |\n",
    "| loader   | container_name | das-cli-loader        |\n",
    "| openfaas | container_name | das-cli-openfaas-8080 |\n",
    "+----------+----------------+-----------------------+\n",
    "```\n",
    "\n",
    "It's because you already have a config file in `~/.das`. If that's the case you need to decide if you want to re-use the same port numbers or not. It's OK to have several databases in your machine. They are Docker containers listening in the given port.\n",
    "\n",
    "If the previous `das-cli config list` command output is empty, you just need to create a new config file. You can do so by running\n",
    "\n",
    "```\n",
    "das-cli config set\n",
    "```\n",
    "\n",
    "In a terminal. When you have done it, run the next cell to make sure you have a config file in place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05fd08e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "!das-cli config list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cedeebcb",
   "metadata": {},
   "source": [
    "Containers for the DBMS servers and OpenFaas will be created listening on the given ports. Run the next cell to make sure any previously used containers are properly removed. If there are none, nothing will be done."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f1680bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "!das-cli db stop\n",
    "!das-cli faas stop"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57a8fc46",
   "metadata": {},
   "source": [
    "Now we need to start the DBMS servers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbda39b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "!das-cli db start"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f2157ae",
   "metadata": {},
   "source": [
    "You can double check that the DB containers are in place listing the active docker containers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee2f8db7",
   "metadata": {},
   "outputs": [],
   "source": [
    "!docker ps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b9821f2",
   "metadata": {},
   "source": [
    "You should see containers for Redis and MongoDB listening on the ports you defined in the config file.\n",
    "\n",
    "Now we need to load a MeTTa file. You can use your own file here or run the next cell to download the same file we used in [this section](#remotedas)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8882d701",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -o /tmp/.get.output https://raw.githubusercontent.com/singnet/das-metta-parser/master/tests/data/animals.metta && mv -f animals.metta /tmp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70763e3e",
   "metadata": {},
   "source": [
    "You may want to change the path in the cell below to point to another file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96c98178",
   "metadata": {},
   "outputs": [],
   "source": [
    "!das-cli metta load /tmp/animals.metta"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae5d4391",
   "metadata": {},
   "source": [
    "You may call `das-cli metta load` multiple times loading different files. To clear the databases you can use `das-cli db restart`.\n",
    "\n",
    "Once you're done loading the knowledge base, you need to start the FaaS server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "985e6e84",
   "metadata": {},
   "outputs": [],
   "source": [
    "!das-cli faas start"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63a8fdcc",
   "metadata": {},
   "source": [
    "It's done. At this point you should be able to point one or more remote DAS to this DAS Server, as we described [here](#remotedas)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}