Add simulation that probes mempool

[emmanueldenloye]

This PR sets out to create a simulation of chainweb-node that observes changes to the list of pending transactions in a node's mempool. Doing this requires that we add these components to the simulation.

• A transaction generator that allows to both "normal" transactions (like coin transfers) as well as transactions with either big module definitions or high gas costs
• An elasticsearch backend for tracking interesting events during the simulation.
• A set of two executables from the repo chainweb-peers. The first discovers nodes in the simulation and the second pings for the list of pending transactions in each of these nodes mempool and reports the relevant data to the elasticsearch backend.

This is the first part of a larger effort to create a testbed for diagnosing/understanding chainweb-node at the network level.

Here are things that seem to be brittle and work temperamentally :

• The chainweb-peers executable seems to not be able to reliably successfully ping the CUT endpoint of chainweb-node. As such, once this problem is addressed, we should get rid of the cut-checker.nix file.

• The dependency of each process in process-compose setup for the chainweb-peers is not well defined. It shouldn't need to take so long for the services txg and show-es-output to start and produce sound output.

[enobayram]

@emmanueldenloye I'm getting the following error when I try to run chainweb-peers locally:

nix run .#chainweb-peers
error: [json.exception.parse_error.101] parse error at line 2817, column 1: syntax error while parsing object key - invalid literal; last read: '"locked": {<U+000A><'; expected string literal

The same for nix build .#chainweb-peers/container.

[emmanueldenloye]

I've posted two curl commands demonstrating that the elasticesarch indexes holding tx data from the simulation both exist and have data in them.

The first index polled is chainweb-fast-development-txg. The houses data associated with a batch of transactions submitted to the chainweb send endpoint. In particular, when the transactions were sent to the endpoint and when their request keys were returned. The second index fast-development-nodes, shows the pending txs in the mempool of a particular host.

(chainweb-fast-development-txg index)

[emmanuel@nixos:~/work/devnet]$ curl -H 'Content-Type: application/json' -X GET "http://localhost:9200/chainweb-fast-development-txg/_search?pretty"
{
  "took" : 63,
  "timed_out" : false,
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 498,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "chainweb-fast-development-txg",
        "_type" : "_doc",
        "_id" : "-RaMio0B1_XrH3v1k3U7",
        "_score" : 1.0,
        "_source" : {
          "batch-confirmation-time" : 170742648505020,
          "batch-submission-time" : 170742648504859,
          "chainwebVersion" : "fast-development",
          "requestKeys" : [
            "nmURMV1o7RfxL_csQghphwuM0yEGJBVSv8vn2i9TwqQ"
          ],
          "timestamp" : 170742648505027
        }
      },
      {
        "_index" : "chainweb-fast-development-txg",
        "_type" : "_doc",
        "_id" : "_BaMio0B1_XrH3v1lnVu",
        "_score" : 1.0,
        "_source" : {
          "batch-confirmation-time" : 170742648586777,
          "batch-submission-time" : 170742648586359,
          "chainwebVersion" : "fast-development",
          "requestKeys" : [
            "js2dYbo2lMRK6fY35SkFEqIcIW3Ov9yOn4Sg6geIwnI"
          ],
          "timestamp" : 170742648586791
        }
      },
      {
        "_index" : "chainweb-fast-development-txg",
        "_type" : "_doc",
        "_id" : "HxaMio0B1_XrH3v1u3ba",
        "_score" : 1.0,
        "_source" : {
          "batch-confirmation-time" : 170742649544793,
          "batch-submission-time" : 170742649544317,
          "chainwebVersion" : "fast-development",
          "requestKeys" : [
            "qByb_IKUXi945h9EnV9b-fncjeSqKEhU750694qOK-k"
          ],
          "timestamp" : 170742649544807
        }
      },
      {
        "_index" : "chainweb-fast-development-txg",
        "_type" : "_doc",
        "_id" : "IRaMio0B1_XrH3v1vnYA",
        "_score" : 1.0,
        "_source" : {
          "batch-confirmation-time" : 170742649599776,
          "batch-submission-time" : 170742649599359,
          "chainwebVersion" : "fast-development",
          "requestKeys" : [
            "tHk3P7uWSO_C52Zmoi9343CJu1WXjVY0KMsBwjTJcCQ"
          ],
          "timestamp" : 170742649599792
        }
      },
      {
        "_index" : "chainweb-fast-development-txg",
        "_type" : "_doc",
        "_id" : "IhaMio0B1_XrH3v1v3YE",
        "_score" : 1.0,
        "_source" : {
          "batch-confirmation-time" : 170742649625919,
          "batch-submission-time" : 170742649625628,
          "chainwebVersion" : "fast-development",
          "requestKeys" : [
            "-5pnF_IhoNuHlhm2MG7JqYZux_E110nAW1-_9nU9Owg"
          ],
          "timestamp" : 170742649625927
        }
      },
      {
        "_index" : "chainweb-fast-development-txg",
        "_type" : "_doc",
        "_id" : "JRaMio0B1_XrH3v1wnYo",
        "_score" : 1.0,
        "_source" : {
          "batch-confirmation-time" : 170742649706236,
          "batch-submission-time" : 170742649705832,
          "chainwebVersion" : "fast-development",
          "requestKeys" : [
            "sgmZ1c-KSGqIIFDo6uJyHg7vNgZebGyR-7VZSOhtFzE"
          ],
          "timestamp" : 170742649706251
        }
      },
      {
        "_index" : "chainweb-fast-development-txg",
        "_type" : "_doc",
        "_id" : "JxaMio0B1_XrH3v1xHZA",
        "_score" : 1.0,
        "_source" : {
          "batch-confirmation-time" : 170742649759781,
          "batch-submission-time" : 170742649759361,
          "chainwebVersion" : "fast-development",
          "requestKeys" : [
            "KVyqrBx93C8hfPkUp8DxVFTkFNMQfFVyM7qo4yeCScc"
          ],
          "timestamp" : 170742649759796
        }
      },
      {
        "_index" : "chainweb-fast-development-txg",
        "_type" : "_doc",
        "_id" : "KRaMio0B1_XrH3v1xnZh",
        "_score" : 1.0,
        "_source" : {
          "batch-confirmation-time" : 170742649814470,
          "batch-submission-time" : 170742649814262,
          "chainwebVersion" : "fast-development",
          "requestKeys" : [
            "WvL46uNrcAInbuQAfAKuLuhh1E1p_uH3gyYfoZpAfF8"
          ],
          "timestamp" : 170742649814477
        }
      },
      {
        "_index" : "chainweb-fast-development-txg",
        "_type" : "_doc",
        "_id" : "HBaMio0B1_XrH3v1uHaj",
        "_score" : 1.0,
        "_source" : {
          "batch-confirmation-time" : 170742649462552,
          "batch-submission-time" : 170742649462185,
          "chainwebVersion" : "fast-development",
          "requestKeys" : [
            "pIIJOZWTi9JGMb9Z_IUF3tvlUfqBIwQzk6FH-H5RqoI"
          ],
          "timestamp" : 170742649462564
        }
      },
      {
        "_index" : "chainweb-fast-development-txg",
        "_type" : "_doc",
        "_id" : "HRaMio0B1_XrH3v1uXa2",
        "_score" : 1.0,
        "_source" : {
          "batch-confirmation-time" : 170742649490006,
          "batch-submission-time" : 170742649489590,
          "chainwebVersion" : "fast-development",
          "requestKeys" : [
            "yDaD7Hwznr1c4OPMAzIQ_m-a6OYlJcm_w10iOiNhndM"
          ],
          "timestamp" : 170742649490021
        }
      }
    ]
  }
}

(fast-development-nodes index)

[emmanuel@nixos:~/work/devnet]$ curl -H 'Content-Type: application/json' -X GET "http://localhost:9200/fast-development-nodes/_search?pretty"
{
  "took" : 280,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3012,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "fast-development-nodes",
        "_type" : "_doc",
        "_id" : "VRaNio0B1_XrH3v1YXgY",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost:1789",
          "pending-tx" : "uSLpYqIsW91f1wmcz3usO9YQOpipESMoHPzHpd5YyeE",
          "timestamp" : 1.707426537748321E9
        }
      },
      {
        "_index" : "fast-development-nodes",
        "_type" : "_doc",
        "_id" : "VhaNio0B1_XrH3v1YXgY",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost:1789",
          "pending-tx" : "wPtWmqHiFVXQt-ORa6CrDtdG1p1oUXJGu5o3BfG6VHc",
          "timestamp" : 1.707426537748321E9
        }
      },
      {
        "_index" : "fast-development-nodes",
        "_type" : "_doc",
        "_id" : "VxaNio0B1_XrH3v1YXgY",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost:1789",
          "pending-tx" : "S29dm-pIdh-Kb3MVVj-tp0KF64QE80L9nm7Q1F-iJNE",
          "timestamp" : 1.707426537748321E9
        }
      },
      {
        "_index" : "fast-development-nodes",
        "_type" : "_doc",
        "_id" : "WBaNio0B1_XrH3v1YXgY",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost:1789",
          "pending-tx" : "LnhUIq_NtM7DzziuEpG5sBEiCPqXCwI4uBx4hr5zBpg",
          "timestamp" : 1.707426537748321E9
        }
      },
      {
        "_index" : "fast-development-nodes",
        "_type" : "_doc",
        "_id" : "WRaNio0B1_XrH3v1YXgY",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost:1789",
          "pending-tx" : "BmDpwevAbhP_3ZPX85h6m9vCLZYg0uUzvTCOkn1Zvss",
          "timestamp" : 1.707426537748321E9
        }
      },
      {
        "_index" : "fast-development-nodes",
        "_type" : "_doc",
        "_id" : "WhaNio0B1_XrH3v1YXgY",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost:1789",
          "pending-tx" : "u8OZ203fGmEAxY59cCmjmnnSlvV3gaU4qQ28OLWQ8QA",
          "timestamp" : 1.707426537748321E9
        }
      },
      {
        "_index" : "fast-development-nodes",
        "_type" : "_doc",
        "_id" : "WxaNio0B1_XrH3v1YXgY",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost:1789",
          "pending-tx" : "dCU4xen7zsle-0CkggZIXQNnJ32fGdJD5oRhhuz4BKM",
          "timestamp" : 1.707426537748321E9
        }
      },
      {
        "_index" : "fast-development-nodes",
        "_type" : "_doc",
        "_id" : "XBaNio0B1_XrH3v1YXgY",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost:1789",
          "pending-tx" : "hf4mvSaZTd1kNLMcdH7Zfxtv2270iO0JaYpsBl5RznU",
          "timestamp" : 1.707426537748321E9
        }
      },
      {
        "_index" : "fast-development-nodes",
        "_type" : "_doc",
        "_id" : "XRaNio0B1_XrH3v1YXgY",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost:1789",
          "pending-tx" : "s5JUZ3MrFuVAaKE9hbPuYxpQqgEBEvA9Q6pi863iibs",
          "timestamp" : 1.707426537748321E9
        }
      },
      {
        "_index" : "fast-development-nodes",
        "_type" : "_doc",
        "_id" : "XhaNio0B1_XrH3v1YXgY",
        "_score" : 1.0,
        "_source" : {
          "host" : "localhost:1789",
          "pending-tx" : "yrSnMi7_nbSj7tMTXr6RLH9u2jsX9Di4fOs_aLqgnWY",
          "timestamp" : 1.707426537748321E9
        }
      }
    ]
  }
}

[emmanueldenloye]

@emmanueldenloye I'm getting the following error when I try to run chainweb-peers locally:

nix run .#chainweb-peers
error: [json.exception.parse_error.101] parse error at line 2817, column 1: syntax error while parsing object key - invalid literal; last read: '"locked": {<U+000A><'; expected string literal

The same for nix build .#chainweb-peers/container.

I'm posting this here for posterity. The problem is in the flake.lock file. The way to reproduce the error without using nix (in other words, to make sure the flake.lock is valid), you can use the following command:

jq empty flake.lock