1. Network Message Structure.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Where to Start?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we describe Bitcoin to our friends and enemies, we often struggle to decide which aspect to describe first. Bitcoin has many facets and a perfect explanation of any single facet usually tails to do justice to the whole. Here's a few facets of Bitcoin:\n",
    "* Hard money, deflationary monetary policy\n",
    "* Anarchic, ungovernable FOSS project\n",
    "* Programmable money\n",
    "* Payment system\n",
    "* Censorship-resistant P2P file sharing network\n",
    "* A ledger\n",
    "\n",
    "Describing Bitcoin as software is similarly difficult. Here's a few facets of Bitcoin in this context:\n",
    "* An internet protocol atop TCP/IP\n",
    "* Smart contracting system\n",
    "* A \"Blockchain\"\n",
    "* A set of transaction abstractions (TxIn, TxOut, script hashes)\n",
    "\n",
    "This course will begin with the network protocol aspect. I choose this for a few reasons: \n",
    "* Network programming is very fun.\n",
    "* We all use the internet, but don't really understand how it works. Here's our chance to dig in and understand a little more deeply.\n",
    "* We can be \"hands on\" from the very first. We will connect directly to [full nodes](https://bitnodes.earn.com/nodes/) and learn to speak their language.\n",
    "* As we go deeper I will introduce whatever concepts are needed to understand what our peers are telling us. In a sense, this is a very empirical approach!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Connecting to the Bitcoin Network\n",
    "\n",
    "OK! Let's connect to the Bitcoin Network.\n",
    "\n",
    "Since the Bitcoin Network is peer-to-peer, we must find some specific peer to connect to.\n",
    "\n",
    "[This site](https://bitnodes.earn.com/nodes/) has a nice listing of visible nodes in the Bitcoin network. Choose one. Look at the \"address\" column. You should see something like \"35.187.200.6:8333\". This is the \"address\" of the node you've selected. This address is composed of two values: an Internet Protocol (IP) address (e.g. 35.187.200.6), and a port (e.g. 8333) separated by a colon.\n",
    "\n",
    "Paste in the IP and port of the node you selected in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL THESE IN!\n",
    "PEER_IP = \"\"\n",
    "PEER_PORT = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how to open a raw socket connection to an IP:Port pair."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import socket\n",
    "\n",
    "sock = socket.socket()\n",
    "sock.connect((PEER_IP, PEER_PORT))\n",
    "\n",
    "response = sock.recv(1024)\n",
    "\n",
    "print(\"response: \", response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK. We created a socket, which like a tunnel across the internet that behaves like a file. You can write data into a socket using `sock.send(message_bytes)` and read from a socket using `sock.recv(number_of_bytes_to_read)`. Similar to a file.\n",
    "\n",
    "But nothing happened. Actually, the socket is still attempting to receive bytes over the TCP connection. The code is blocked on the line `response = sock.recv(1024)`. This will wait forever until the Bitcoin node at the other end of our socket connection sends us a response.\n",
    "\n",
    "You may have noticed that to the left of every Jupyter cell is the text `In [ ]:` if the cell hasn't been executed yet, and somthing like `In [7]:` if the cell was the 7th cell executed. But the cell above says `In [*]:`. This means that it's still executing. The code is stuck. Press `escape ii` or hit the ■ button in the menu above to tell Jupyter to kill the process in the cell above.\n",
    "\n",
    "The reason the code is stuck is because we didn't properly introduce ourselves to our peer. With Bitcoin, we must perform a [Version Handshake](https://en.bitcoin.it/wiki/Version_Handshake) in order to begin exchanging messages.\n",
    "\n",
    "So let's try again. I'm going to give you a `VERSION` bytestring without telling you how I came up with it. Before calling `sock.recv(1024)` we will first call `sock.send(VERSION)` because the Bitcoin Version Handshake demands that the node which initiates the connection send the first `version` message."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import socket\n",
    "\n",
    "PEER_IP = \"35.187.200.6\"\n",
    "\n",
    "PEER_PORT = 8333\n",
    "\n",
    "# magic \"version\" bytestring\n",
    "VERSION = b'\\xf9\\xbe\\xb4\\xd9version\\x00\\x00\\x00\\x00\\x00j\\x00\\x00\\x00\\x9b\"\\x8b\\x9e\\x7f\\x11\\x01\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x93AU[\\x00\\x00\\x00\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00rV\\xc5C\\x9b:\\xea\\x89\\x14/some-cool-software/\\x01\\x00\\x00\\x00\\x01'\n",
    "\n",
    "sock = socket.socket()\n",
    "sock.connect((PEER_IP, PEER_PORT))\n",
    "\n",
    "# initiate the \"version handshake\"\n",
    "sock.send(VERSION)\n",
    "\n",
    "# receive their \"version\" response\n",
    "response = sock.recv(1024)\n",
    "\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code no longer gets stuck at `response = sock.recv(1024)` because the peer answered our `version` message with their own `version` message. We're learning to say \"hello\" in the language of the Bitcoin network protocol!\n",
    "\n",
    "This table in the [protocol documentation](https://en.bitcoin.it/wiki/Protocol_documentation#version) tells us what information the `version` message contains and how to decipher it. But before we can read the version message specifically (as opposed to the other 26 types), we need to learn to read a Bitcoin protocol message generally. This [\"message structure\"](https://en.bitcoin.it/wiki/Protocol_documentation#Message_structure) table tells us how.\n",
    "\n",
    "The \"description\" and \"comments\" columns tell us what each row in the table means. The \"field size\" column tell us the number of bytes each field takes up, and the \"data type\" column tells us how we should interpret these bytes -- e.g. whether they a number, a string, a list etc.\n",
    "\n",
    "Here's the value I received when I ran the previous cell:\n",
    "\n",
    "```\n",
    "b'\\xf9\\xbe\\xb4\\xd9version\\x00\\x00\\x00\\x00\\x00f\\x00\\x00\\x00\\xc6\\xa7\\xa2(\\x7f\\x11\\x01\\x00\\r\\x04\\x00\\x00\\x00\\x00\\x00\\x00&\\xffT[\\x00\\x00\\x00\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\r\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00$\\xfa\\t\\n\\xc5y\\xf7\\t\\x10/Satoshi:0.16.0/\\xae\"\\x08\\x00\\x01'\n",
    "```\n",
    "\n",
    "Here's how to read the 5 fields in the [\"message structure\"](https://en.bitcoin.it/wiki/Protocol_documentation#Message_structure) table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "version_bytes = b'\\xf9\\xbe\\xb4\\xd9version\\x00\\x00\\x00\\x00\\x00f\\x00\\x00\\x00\\xc6\\xa7\\xa2(\\x7f\\x11\\x01\\x00\\r\\x04\\x00\\x00\\x00\\x00\\x00\\x00&\\xffT[\\x00\\x00\\x00\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\r\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00$\\xfa\\t\\n\\xc5y\\xf7\\t\\x10/Satoshi:0.16.0/\\xae\"\\x08\\x00\\x01'\n",
    "\n",
    "print('4 \"magic\" bytes: ', version_bytes[:4])\n",
    "print('12 \"command\" bytes: ', version_bytes[4:4+12])\n",
    "payload_length_bytes = version_bytes[4+12:4+12+4]\n",
    "# here I actually interpret these bytes as an integer\n",
    "# more on this later!\n",
    "payload_length = int.from_bytes(payload_length_bytes, 'little')\n",
    "print('4 \"length\" bytes', payload_length_bytes)\n",
    "print('4 \"checksum\" bytes', version_bytes[4+12+4:4+12+4+4])\n",
    "print(payload_length, ' \"payload\" bytes', version_bytes[4+12+4+4:4+12+4+4+payload_length])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's ugly as sin, but by simply [slicing](https://stackoverflow.com/questions/509211/understanding-pythons-slice-notation) the `version_bytes` we can extract:\n",
    "\n",
    "* `version`: first 4 bytes\n",
    "* `command`: next 12 bytes\n",
    "* `length`: next 4 bytes\n",
    "* `checksum`: next 4 bytes\n",
    "* `payload`: next `length` bytes\n",
    "\n",
    "While indexing is conceptualy simple, it's confusing and fragile.\n",
    "\n",
    "Can't we agree that a better way would be some function like `read(version_bytes, n)` which would just read the next n bytes? Kind of like how we read from the socket earlier: `sock.recv(1024)`. This way we wouldn't need to keep track of how many bytes we've already read (e.g. `4+12+4+4`).\n",
    "\n",
    "Well I have good news. The Python Standard Library contains a [io.BytesIO](https://docs.python.org/3/library/io.html#io.BytesIO) class that does exactly this. \n",
    "\n",
    "The only problem is that we need to call `instance.read` to read from an `io.BytesIO` instance and `instance.recv` to read from a `socket.socket` instance. Since we want all our code to work with sockets, we can create a stupid little `FakeSocket` helper class which can keep our code socket-compatibile while we develop it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from io import BytesIO\n",
    "\n",
    "class FakeSocket:\n",
    "    \n",
    "    def __init__(self, bytes_):\n",
    "        self.stream = BytesIO(bytes_)\n",
    "        \n",
    "    def recv(self, n):\n",
    "        return self.stream.read(n)\n",
    "    \n",
    "    \n",
    "def create_version_socket(skip_bytes=0):\n",
    "    version_bytes = b'\\xf9\\xbe\\xb4\\xd9version\\x00\\x00\\x00\\x00\\x00f\\x00\\x00\\x00\\xc6\\xa7\\xa2(\\x7f\\x11\\x01\\x00\\r\\x04\\x00\\x00\\x00\\x00\\x00\\x00&\\xffT[\\x00\\x00\\x00\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\r\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00$\\xfa\\t\\n\\xc5y\\xf7\\t\\x10/Satoshi:0.16.0/\\xae\"\\x08\\x00\\x01'[skip_bytes:]\n",
    "    return FakeSocket(version_bytes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sock = create_version_socket()\n",
    "\n",
    "print('4 \"magic\" bytes: ', sock.recv(4))\n",
    "print('12 \"command\" bytes: ', sock.recv(12))\n",
    "payload_length_bytes = sock.recv(4)\n",
    "# here I actually interpret these bytes as an integer\n",
    "# more on this later!\n",
    "payload_length = int.from_bytes(payload_length_bytes, 'little')\n",
    "print('4 \"length\" bytes', payload_length_bytes)\n",
    "print('4 \"checksum\" bytes', sock.recv(4))\n",
    "print(payload_length, ' \"payload\" bytes', sock.recv(payload_length))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`sock.recv(payload_length)` is a lot easier to read than `version_bytes[4+12+4+4:4+12+4+4+payload_length]`, wouldn't you say?\n",
    "\n",
    "This also fixes a huge bug that exists the line of code above: `response = sock.recv(1024)`.\n",
    "\n",
    "This statement reads a fixed number of bytes no matter what. 1024 bytes will probably never be enough to read a `block` message, but it's often too much for a `version` message. For example, if you try running the first socket example with enough peers you will eventually see something like this: \n",
    "\n",
    "```\n",
    "b'\\xf9\\xbe\\xb4\\xd9version\\x00\\x00\\x00\\x00\\x00f\\x00\\x00\\x00f\\x9a\\xe5\\x06\\x7f\\x11\\x01\\x00\\r\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\xb9kV[\\x00\\x00\\x00\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\r\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x92\\xed.\\xd6\\xba\\x90\\xa8\\t\\x10/Satoshi:0.16.0/d#\\x08\\x00\\x01\\xf9\\xbe\\xb4\\xd9verack\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00]\\xf6\\xe0\\xe2'\n",
    "```\n",
    "\n",
    "If you look closely you may notice that the magic network string `\\xf9\\xbe\\xb4\\xd9` (explained in next cells) appears twice. `version` follows the first occurrence, and `verack` follows the second. This is not one message! It's two!\n",
    "\n",
    "By always reading exactly the number of bytes we need -- as in the cell above -- we can avoid this problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Read Network Magic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The protocol documentation from earlier lists `0xD9B4BEF9` as the `magic` value for Bitcoin's \"main\" network. This integer (represented in hexidecimal / base 16) identifies TCP packets as Bitcoin network packets. To know whether your program is working with a Bitcoin protocol message it is sufficient to check whether the first 4 bytes of the message are equivalent to the integer `0xD9B4BEF9`.\n",
    "\n",
    "Furthermore, the protocol docs state that this value is \"sent over the network\" as `F9 BE B4 D9`. Notice how similar this is to the `\\xf9\\xbe\\xb4\\xd9` value printed above. If you take `\\xf9\\xbe\\xb4\\xd9`, replace `\\x`'s with spaces and then uppercase it, you have `F9 BE B4 D9`.\n",
    "\n",
    "If you take `\\xf9\\xbe\\xb4\\xd9'`, reverse the groups of bytes (`\\xd9\\xb4\\xbe\\xf9`), strip the `\\x` byte prefixes (`d9b4bef9`), uppercase it (`D9B4BEF9`), and prepend with `0x` prefix to indiciate it's a hexidecimal number (`0xD9B4BEF9`) we arrive at the Bitcoin's magic integer.\n",
    "\n",
    "I expect you're extremely confused right now. I, too, was extremely confused when I first encountered this. But I hope you can also see that there's some pattern to the madness ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hexidecimal integers copied from protocol documentation\n",
    "NETWORK_MAGIC = 0xD9B4BEF9\n",
    "TESTNET_NETWORK_MAGIC = 0x0709110B\n",
    "\n",
    "# notice this looks like any old integer even though we declared it\n",
    "# using a hexidecimal notation. Under the hood Python stores every integer\n",
    "# as base 2 and doesn't care what base integers were initialized in.\n",
    "print(\"NETWORK_MAGIC:\", NETWORK_MAGIC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bytes_to_int(b):\n",
    "    return int.from_bytes(b, 'little')\n",
    "\n",
    "def read_magic(sock):\n",
    "    magic_bytes = sock.recv(4)\n",
    "    magic = bytes_to_int(magic_bytes)\n",
    "    return magic\n",
    "    \n",
    "def is_mainnet_msg(sock):\n",
    "    magic = read_magic(sock)\n",
    "    return magic == NETWORK_MAGIC\n",
    "\n",
    "def is_testnet_msg(sock):\n",
    "    magic = read_magic(sock)\n",
    "    return magic == TESTNET_NETWORK_MAGIC\n",
    "\n",
    "\n",
    "print(\"Mainnet version message?\", is_mainnet_msg(create_version_socket()))\n",
    "print(\"Testnet version message?\", is_testnet_msg(create_version_socket()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above code is pretty straightforward, with two exceptions:\n",
    "\n",
    "* hexidecimal numbers: `0xD9B4BEF9` and `0x0709110B`\n",
    "* bytes -> int conversion: `int.from_bytes(b, 'little')`\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hexidecimal \n",
    "\n",
    "In Python you prefix a number with `0x` to tell the interpreter to interpret it as hexidecimal (base 16), `0o` for octal (base 8), `0b` for binary (base 2). But once you've created the integer, Python forgets what base you used to create it. `17`, `0b10001`, `0o21`, and `0x11` are all equivalent. \n",
    "\n",
    "There are built-in `bin`, `oct`, and `hex` function that take an integer and give you a string representation of the integer in a different base. But they don't actually change the integer because under the hood all integers are stored as base 2 in Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 17\n",
    "\n",
    "print(n, \"is:\")\n",
    "print(bin(n),\"in binary.\")\n",
    "print(oct(n),\"in octal.\")\n",
    "print(hex(n),\"in hexadecimal.\")\n",
    "\n",
    "# These just print out the string representation of n in a different base\n",
    "print(\"type of bin()/oct()/hex()?\", type(bin(n)), type(oct(n)), type(hex(n)))\n",
    "\n",
    "print(\"17 == 0b10001 == 0o21 == 0x11?\", 17 == 0b10001 == 0o21 == 0x11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### bytes -> int\n",
    "\n",
    "Time to talk \"endianness\". Yes, \"endianness\".\n",
    "\n",
    "The Internet Protocol is just a system to send packets of bytes to an address. But there are two ways to send the bytes:\n",
    "\n",
    "> In big-endian format, whenever addressing memory or sending/storing words bytewise, the most significant byte—the byte containing the most significant bit—is stored first (has the lowest address) or sent first, then the following bytes are stored or sent in decreasing significance order, with the least significant byte—the one containing the least significant bit—stored last (having the highest address) or sent last.\n",
    "\n",
    "> Little-endian format reverses this order: the sequence addresses/sends/stores the least significant byte first (lowest address) and the most significant byte last (highest address). Most computer systems prefer a single format for all its data; using the system's native format is automatic. But when reading memory or receiving transmitted data from a different computer system, it is often required to process and translate data between the preferred native endianness format to the opposite format.\n",
    "\n",
    "(from [Wikipedia](https://en.wikipedia.org/wiki/Endianness))\n",
    "\n",
    "If we look once again at the magic values chart from the protocol documentation:\n",
    "\n",
    "![image](./images/magic-values.png)\n",
    "\n",
    "You see that the most significant byte for the main network magic is `D9`. This is just how we write numbers: the biggest digit is to the left.\n",
    "\n",
    "But it's sent over the wire as `F9 BE B4 D9` -- now the biggest / most-significant byte is to the right and the smallest / least-significant byte (`F9`) is to the left. This is because it's been encoded \"little-endian\". Now our `bytes_to_int` definition should make a little more sense:\n",
    "\n",
    "```\n",
    "def bytes_to_int(b):\n",
    "    return int.from_bytes(b, 'little')\n",
    "```\n",
    "\n",
    "We're just taking a bytestring and converting to an integer, and python demands we specify the byteorder ('little'). [Here's the relevant section of the Python documentation](https://docs.python.org/3/library/stdtypes.html#int.from_bytes).\n",
    "\n",
    "Try creating another cell in this Jupyter Notebook (Insert > Cell Below, or type \"escape\" then \"b\") and running `int.from_bytes('\\x00')`. You will get an error because Python demands you specify the byteorder -- whether the bytes are little- or big-endian encoded.\n",
    "\n",
    "\n",
    "Another expression you might try to evaluate is `int.from_bytes(b\"\\x00\\x01\", \"little\") == int.from_bytes(b\"\\x01\\x00\", \"big\")`. The beauty of Jupyter Notebook is that you can always create a new cell and test to see whether I'm lying!\n",
    "\n",
    "\n",
    "If you're confused, don't despair. We'll be doing a TON of little-endian decoding and encoding. We'll even do some big-endian encoding because Satoshi didn't make up his mind and stick to a consistent byteorder. In particular, public keys, IP and port numbers are encoded big-endian."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Read `command`\n",
    "\n",
    "Let's try to parse the 12 byte \"command\" section of the message\n",
    "\n",
    "![image](./images/message-structure.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_command(sock):\n",
    "    raw = sock.recv(12)\n",
    "    # remove empty bytes\n",
    "    command = raw.replace(b\"\\x00\", b\"\")\n",
    "    return command\n",
    "    \n",
    "def is_version_msg(sock):\n",
    "    command = read_command(sock)\n",
    "    return b\"version\" == command\n",
    "    \n",
    "def is_verack_msg(sock):\n",
    "    command = read_command(sock)\n",
    "    return b\"verack\" == command\n",
    "\n",
    "# Throw away the first 4 bytes (the magic)\n",
    "sock = create_version_socket(skip_bytes=4)\n",
    "command = read_command(sock)\n",
    "print(\"Command: \", command)\n",
    "\n",
    "sock = create_version_socket(skip_bytes=4)\n",
    "print(\"Is it a 'version' message?\", is_version_msg(sock))\n",
    "\n",
    "sock = create_version_socket(skip_bytes=4)\n",
    "print(\"Is it a 'verack' message?\", is_verack_msg(sock))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Message Payload\n",
    "\n",
    "Let's try to parse the 3 payload-related portions of the message: \"length\", \"checksum\", and \"payload\"\n",
    "\n",
    "![image](./images/message-structure.png)\n",
    "\n",
    "The whole goal of these three attributes is to read the payload -- which could be a newly mined block, or a transaction, or the address of a newly connected bitcoin full node -- and verify that the payload wasn't corrupted at all in transmission over the internet.\n",
    "\n",
    "Payloads vary in length. A [`verack` message](https://en.bitcoin.it/wiki/Protocol_documentation#verack) has empty payload. A [`block` message](https://en.bitcoin.it/wiki/Protocol_documentation#block) payload may contain thousands of transactions, each with tens of inputs and outputs.\n",
    "\n",
    "To deal with the varying lengths of message payloads, the Bitcoin protocol messages always include a `length` parameter which tells us exactly how many bytes to read. This helps us avoid only reading part of the payload and stopping, or overshooting and reading into the next message (like the example above where I read two messages by accident).\n",
    "\n",
    "And once we read the payload, how can we be sure that what we receive is the same as what our peer node sent us?\n",
    "\n",
    "Well, TCP ensures against packet loss using checksums and the Bitcoin network protocol is just a thin layer on top of TCP. [But apparently its checksumming mechanism (which I don't understand) may be less effective than `sha256`](https://bitcoin.stackexchange.com/questions/22882/what-is-the-function-of-the-payload-checksum-field-in-the-bitcoin-protocol). In any case, Satoshi implemented a further [\"checksum\"](https://en.wikipedia.org/wiki/Checksum) to guarantee data integrity.\n",
    "\n",
    "Checksums are a simple idea: You take the message and run it through a [hashing algorithm](https://blog.jscrambler.com/hashing-algorithms/) -- in this case you run it through `sha256` twice -- and take just a few bytes of the result -- in this case the first 4. Given an input `x`, a hashing algorithm `h` will always produce the same output `h(x)`. Since the output is the same, the first four digits of the output will always be the same. And since the key property of hashing functions is going from `h(x) -> x` requires brute forcing, you'd need to try about 256^4 ≈ 500,000,000 values (a byte contains 256 possible values and there are 4 bytes) on average to fake any given payload without modifying the checksum. So I guess Satoshi thought that would help!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hashlib import sha256\n",
    "\n",
    "def read_length(sock):\n",
    "    raw = sock.recv(4)\n",
    "    length = bytes_to_int(raw)\n",
    "    return length\n",
    "\n",
    "def read_checksum(sock):\n",
    "    # FIXME: protocol documentation says this should be an integer ...\n",
    "    raw = sock.recv(4)\n",
    "    return raw\n",
    "\n",
    "def calculate_checksum(payload_bytes):\n",
    "    \"\"\"First 4 bytes of sha256(sha256(payload))\"\"\"\n",
    "    first_round = sha256(payload_bytes).digest()\n",
    "    second_round = sha256(first_round).digest()\n",
    "    first_four_bytes = second_round[:4]\n",
    "    return first_four_bytes\n",
    "\n",
    "def read_payload(sock, length):\n",
    "    payload = sock.recv(length)\n",
    "    return payload\n",
    "\n",
    "# skip the \"magic\" and \"command\" bytes\n",
    "sock = create_version_socket(skip_bytes=4+12)\n",
    "\n",
    "length = read_length(sock)\n",
    "checksum = read_checksum(sock)\n",
    "payload = read_payload(sock, length)\n",
    "\n",
    "print(\"Length: \", length)\n",
    "\n",
    "print(\"Checksum: \", checksum)\n",
    "\n",
    "print(\"Payload: \", payload)\n",
    "\n",
    "print(\"checksum == calculate_checksum(payload)?: \", \n",
    "      checksum == calculate_checksum(payload))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And there we go! We know how to parse all from a Bitcoin protocol message!\n",
    "\n",
    "Lastly, let's create a class that abstracts a Bitcoin protocol message."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Message:\n",
    "    \n",
    "    def __init__(self, command, payload):\n",
    "        self.command = command\n",
    "        self.payload = payload\n",
    "\n",
    "    @classmethod\n",
    "    def read_from_socket(cls, sock):\n",
    "        magic = read_magic(sock)\n",
    "        if magic != NETWORK_MAGIC:\n",
    "            raise ValueError('Network magic is wrong')\n",
    "\n",
    "        command = read_command(sock)\n",
    "        payload_length = read_length(sock)\n",
    "        checksum = read_checksum(sock)\n",
    "        payload = read_payload(sock, payload_length)\n",
    "        \n",
    "        calculated_checksum = calculate_checksum(payload)\n",
    "        if calculated_checksum != checksum:\n",
    "            raise RuntimeError(\"Checksums don't match\")\n",
    "\n",
    "        if payload_length != len(payload):\n",
    "            raise RuntimeError(\"Tried to read {payload_length} bytes, only received {len(payload)} bytes\")\n",
    "\n",
    "        return cls(command, payload)\n",
    "\n",
    "    def serialize(self):\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f\"<Message command={self.command} payload={self.payload}>\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import socket\n",
    "\n",
    "PEER_IP = \"35.187.200.6\"\n",
    "\n",
    "PEER_PORT = 8333\n",
    "\n",
    "# magic \"version\" bytestring\n",
    "VERSION = b'\\xf9\\xbe\\xb4\\xd9version\\x00\\x00\\x00\\x00\\x00j\\x00\\x00\\x00\\x9b\"\\x8b\\x9e\\x7f\\x11\\x01\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x93AU[\\x00\\x00\\x00\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00rV\\xc5C\\x9b:\\xea\\x89\\x14/some-cool-software/\\x01\\x00\\x00\\x00\\x01'\n",
    "\n",
    "sock = socket.socket()\n",
    "sock.connect((PEER_IP, PEER_PORT))\n",
    "\n",
    "# initiate the \"version handshake\"\n",
    "sock.send(VERSION)\n",
    "\n",
    "# receive their \"version\" response\n",
    "msg = Message.read_from_socket(sock)\n",
    "\n",
    "print(msg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There you go. You now know how to read a Bitcoin message directly from a raw socket, verify that it is structured correctly, and interpret it as an instance of a Python `Message` class.\n",
    "\n",
    "You are almost ready to do the [Version Handshake](https://en.bitcoin.it/wiki/Version_Handshake) unassisted. We just need to learn how to read and create `version` and `verack` payloads. We'll cover this in the next lesson!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}