From b6c3efa4b03cdd5a6c654a0b4cb855afc53084a6 Mon Sep 17 00:00:00 2001 From: Casian Lacatusu Date: Fri, 15 Nov 2019 16:25:47 +0200 Subject: [PATCH 1/4] create docker-compose; update readme --- README.md | 98 ++++++++---- docker-compose-explorer.yml | 43 +++++ ...r-compose.yml => docker-compose-single.yml | 0 lite-explorer.config.json | 149 ++++++++++++++++++ 4 files changed, 256 insertions(+), 34 deletions(-) create mode 100644 docker-compose-explorer.yml rename docker-compose.yml => docker-compose-single.yml (100%) create mode 100644 lite-explorer.config.json diff --git a/README.md b/README.md index 9bf5c49..718efc2 100644 --- a/README.md +++ b/README.md @@ -4,23 +4,25 @@ Memento is a tool developed by Alethio to scrape and index data from any web3-co The main goal of the tool is to scrape the raw data from the network, do the necessary transformations and insert it into an indexed database from which it is exposed through the integrated API. -Seamless integration with the [Ethereum Lite Explorer by Alethio](https://github.com/Alethio/ethereum-lite-explorer) is coming soon to provide you the full blockchain exploration capabilities without the need of a third party. +Seamless integration with the [Ethereum Lite Explorer by Alethio](https://github.com/Alethio/ethereum-lite-explorer) provides you the full blockchain exploration capabilities without the need of a third party. -Easily check the system status, perform various actions and manage your configuration through the built-in dashboard. +Easily check the system status, perform various actions and manage your configuration through the built-in dashboard. ![memento dashboard](/web/assets/images/preview.png "Memento dashboard") ## Contents - [Memento - Ethereum scraper and indexer](#memento---ethereum-scraper-and-indexer) + - [Contents](#contents) - [Features](#features) - [Getting started](#getting-started) - [Configuration](#configuration) - [Via dashboard](#via-dashboard) - [Via config file / command line arguments](#via-config-file--command-line-arguments) - - [Installation](#installation) - - [Building from source](#building-from-source) - - [Running with docker-compose](#running-with-docker-compose) + - [Running](#running) + - [Memento with lite explorer](#memento-with-lite-explorer) + - [Standalone via docker compose](#standalone-via-docker-compose) - [Running with Docker](#running-with-docker) + - [Building from source](#building-from-source) - [Example output](#example-output) - [Result](#result) - [Example setups](#example-setups) @@ -107,57 +109,51 @@ REDIS_PASSWORD PG_PASSWORD ``` -### Installation -#### Building from source -**Prerequisites** -- a working Golang environment (tested with go v1.12.9) - - requires go modules (`>=go v1.11`) -- Postgres -- Redis >= 5.0 -- A JSON-RPC enabled and accessible Ethereum Client (Infura also works) +### Running -**Clone the repo** -```shell script -git clone git@github.com:Alethio/memento.git -cd memento -``` +#### Memento with [lite explorer](https://github.com/Alethio/ethereum-lite-explorer) +We've included a [docker-compose](./docker-compose-explorer.yml) that packages a full working environment with Memento and the [Ethereum Lite Explorer by Alethio](https://github.com/Alethio/ethereum-lite-explorer). -**Build the executable** (it uses go modules, so it will automatically download the dependencies): +**Copy the config and edit it as needed.** By default, the values for postgres and redis are correct for using with docker compose. ```shell script -make -``` +mkdir -p .volumes/memento +mkdir -p .volumes/lite-explorer -**Copy the sample configuration** and do the necessary adjustments -```shell script -cp config-sample.yml config.yml +cp config-sample.yml .volumes/memento/config.yml +cp lite-explorer.config.json .volumes/lite-explorer/config.json ``` -**Start scraping** :tada: +**Start everything** ```shell script -./memento run --vv +docker-compose up -d ``` -**Open the dashboard to check progress** +**Open the dashboard** to check indexing progress ``` http://localhost:3000 ``` -#### Running with docker-compose +**Open the Lite Explorer** and start exploring :tada: +``` +http://localhost:80 +``` + +#### Standalone via docker compose The simplest way to run the whole setup is by using the included docker compose -Copy the config and edit it as needed. By default, the values for postgres and redis are correct for using with docker compose. +**Copy the config and edit it as needed.** By default, the values for postgres and redis are correct for using with docker compose. ```shell script mkdir -p .volumes/memento cp config-sample.yml .volumes/memento/config.yml ``` -Start everything +**Start everything** ```shell script docker-compose up -d ``` -Open the dashboard to check progress +**Open the dashboard to check progress** ``` http://localhost:3000 ``` @@ -165,19 +161,53 @@ http://localhost:3000 #### Running with Docker If you already have a postgres instance & a redis instance set up and still want the simplest way, you can use the docker image from Dockerhub. -Copy the config and edit it as needed, making sure Memento is able to connect to postgres, redis and the Ethereum client of choice. +**Copy the config and edit it as needed**, making sure Memento is able to connect to postgres, redis and the Ethereum client of choice. ```shell script mkdir -p .volumes/memento cp config-sample.yml .volumes/memento/config.yml ``` -Start memento in Docker +**Start memento in Docker** ```shell script docker run --name memento -d -v /path/to/config/folder:/config/ -p 3000:3000 -p 3001:3001 alethio/memento:latest ``` -Open the dashboard to check progress +**Open the dashboard to check progress** +``` +http://localhost:3000 +``` + +### Building from source +**Prerequisites** +- a working Golang environment (tested with go v1.12.9) + - requires go modules (`>=go v1.11`) +- Postgres +- Redis >= 5.0 +- A JSON-RPC enabled and accessible Ethereum Client (Infura also works) + +**Clone the repo** +```shell script +git clone git@github.com:Alethio/memento.git +cd memento +``` + +**Build the executable** (it uses go modules, so it will automatically download the dependencies): +```shell script +make +``` + +**Copy the sample configuration** and do the necessary adjustments +```shell script +cp config-sample.yml config.yml +``` + +**Start scraping** :tada: +```shell script +./memento run --vv +``` + +**Open the dashboard to check progress** ``` http://localhost:3000 ``` diff --git a/docker-compose-explorer.yml b/docker-compose-explorer.yml new file mode 100644 index 0000000..3a0c900 --- /dev/null +++ b/docker-compose-explorer.yml @@ -0,0 +1,43 @@ +version: '3.7' +services: + redis: + container_name: memento-redis + image: redis:5.0.5-alpine + restart: always + command: ["redis-server", "--appendonly", "yes"] + volumes: + - .volumes/redis:/data + postgres: + container_name: memento-postgres + image: postgres:11.5-alpine + restart: always + environment: + - POSTGRES_USER=memento + - POSTGRES_DB=memento + - POSTGRES_PASSWORD=password + volumes: + - .volumes/postgresql:/var/lib/postgresql/data + memento: + container_name: memento + image: alethio/memento:latest + restart: always + depends_on: + - redis + - postgres + ports: + - 127.0.0.1:3001:3001 + - 127.0.0.1:3000:3000 + environment: + - PG_PASSWORD=password + volumes: + - .volumes/memento:/config + explorer: + container_name: memento-lite-explorer + image: alethio/ethereum-lite-explorer:latest + restart: always + depends_on: + - memento + ports: + - 127.0.0.1:80:80 + volumes: + - .volumes/lite-explorer/config.json:/usr/share/nginx/html/config.json diff --git a/docker-compose.yml b/docker-compose-single.yml similarity index 100% rename from docker-compose.yml rename to docker-compose-single.yml diff --git a/lite-explorer.config.json b/lite-explorer.config.json new file mode 100644 index 0000000..36c0aba --- /dev/null +++ b/lite-explorer.config.json @@ -0,0 +1,149 @@ +{ + "GOOGLE_ANALYTICS_ID": null, + "HOTJAR_ID": null, + "SENTRY_ENABLED": false, + + "pluginsUrl": "plugins", + + "plugins": { + "plugin://aleth.io/eth-common?v=2.4.0": { + + }, + "plugin://aleth.io/3box?v=1.1.0": { + "ipfsUrlMask": "https://ipfs.infura.io/ipfs/%s" + }, + "plugin://aleth.io/eth-memento?v=1.1.0": { + "apiBasePath": "http://localhost:3001/api/explorer" + } + }, + + "pages": [{ + "def": "page://aleth.io/block", + "children": { + "sidebar": [{ + "def": "context://aleth.io/block/list", + "children": [ + { "def": "module://aleth.io/block/list" } + ] + }], + "content": [{ + "def": "module://aleth.io/memento/block/basic", + "pageCritical": true, + "children": { + "confirmations": [ + { "def": "module://aleth.io/block/confirmations" } + ] + } + }, { + "def": "module://aleth.io/memento/block/txs", + "pageCritical": true + }, { + "def": "module://aleth.io/memento/block/advanced", + "pageCritical": true + }, { + "def": "module://aleth.io/memento/block/logs-bloom", + "pageCritical": true + }] + } + }, { + "def": "page://aleth.io/uncle", + "children": { + "content": [{ + "def": "module://aleth.io/memento/uncle/details", + "pageCritical": true + }] + } + }, { + "def": "page://aleth.io/tx", + "children": { + "sidebar": [{ + "def": "context://aleth.io/memento/tx/parentBlock", + "children": [{ + "def": "module://aleth.io/tx/list" + }] + }], + "content": [{ + "def": "context://aleth.io/memento/tx/parentBlock", + "pageCritical": true, + "children": [{ + "def": "module://aleth.io/memento/tx/basic", + "children": { + "blockConfirmations": [{ + "def": "context://aleth.io/memento/tx/parentBlock", + "children": [{ + "def": "module://aleth.io/block/confirmations" + }] + }] + } + }, { + "def": "module://aleth.io/memento/tx/advanced" + }, { + "def": "module://aleth.io/memento/tx/summary" + }, { + "def": "module://aleth.io/memento/tx/payload" + }] + }] + } + }, { + "def": "page://aleth.io/account", + "children": { + "identicon": [{ + "def": "module://aleth.io/3box/profile", + "children": { + "fallback": [{ "def": "module://aleth.io/account/identicon" }] + } + }], + "top": [ + { "def": "module://aleth.io/memento/account/details", "pageCritical": true } + ], + "bottom": [ + { "def": "module://aleth.io/memento/account/txs" }, + { "def": "module://aleth.io/memento/account/contract" } + ] + } + }, { + "def": "page://aleth.io/dashboard", + "children": { + "content": [ + { "def": "module://aleth.io/search" }, + { + "def": "context://aleth.io/dashboard/latestBlockRange", + "children": [ + { + "def": "module://aleth.io/dashboard/charts", + "children": { + "content": [{ + "def": "module://aleth.io/dashboard/blocksChart", + "children": { + "children": [ + { + "def": "context://aleth.io/block/latest", + "children": [ + { "def": "module://aleth.io/dashboard/latestBlockInfo" } + ] + } + ] + } + }] + } + } + ] + } + ] + } + }], + + "rootModules": { + "toolbarTop": [{ + "def": "module://aleth.io/toolbar/search" + }], + "toolbarBottom": [{ + "def": "module://aleth.io/toolbar/feedback" + }], + "topbar": [{ + "def": "module://aleth.io/topbar/search" + }, { + "def": "module://aleth.io/topbar/feedback" + }] + } +} From 3c9b86466b13519776331ba22a792691f8c9dd74 Mon Sep 17 00:00:00 2001 From: Casian Lacatusu Date: Fri, 15 Nov 2019 17:05:55 +0200 Subject: [PATCH 2/4] update reamde --- README.md | 24 +++++++++---------- ...r-compose-single.yml => docker-compose.yml | 0 2 files changed, 12 insertions(+), 12 deletions(-) rename docker-compose-single.yml => docker-compose.yml (100%) diff --git a/README.md b/README.md index 718efc2..fcf57f6 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ Easily check the system status, perform various actions and manage your configur - [Via dashboard](#via-dashboard) - [Via config file / command line arguments](#via-config-file--command-line-arguments) - [Running](#running) - - [Memento with lite explorer](#memento-with-lite-explorer) + - [Memento with Lite Explorer](#memento-with-lite-explorerhttpsgithubcomalethioethereum-lite-explorer) - [Standalone via docker compose](#standalone-via-docker-compose) - [Running with Docker](#running-with-docker) - [Building from source](#building-from-source) @@ -111,10 +111,10 @@ PG_PASSWORD ### Running -#### Memento with [lite explorer](https://github.com/Alethio/ethereum-lite-explorer) +#### Memento with [Lite Explorer](https://github.com/Alethio/ethereum-lite-explorer) We've included a [docker-compose](./docker-compose-explorer.yml) that packages a full working environment with Memento and the [Ethereum Lite Explorer by Alethio](https://github.com/Alethio/ethereum-lite-explorer). -**Copy the config and edit it as needed.** By default, the values for postgres and redis are correct for using with docker compose. +Copy the config and edit it as needed. By default, the values for postgres and redis are correct for using with docker compose. ```shell script mkdir -p .volumes/memento mkdir -p .volumes/lite-explorer @@ -123,17 +123,17 @@ cp config-sample.yml .volumes/memento/config.yml cp lite-explorer.config.json .volumes/lite-explorer/config.json ``` -**Start everything** +Start everything ```shell script docker-compose up -d ``` -**Open the dashboard** to check indexing progress +Open the dashboard to check indexing progress ``` http://localhost:3000 ``` -**Open the Lite Explorer** and start exploring :tada: +Open the Lite Explorer and start exploring :tada: ``` http://localhost:80 ``` @@ -141,19 +141,19 @@ http://localhost:80 #### Standalone via docker compose The simplest way to run the whole setup is by using the included docker compose -**Copy the config and edit it as needed.** By default, the values for postgres and redis are correct for using with docker compose. +Copy the config and edit it as needed. By default, the values for postgres and redis are correct for using with docker compose. ```shell script mkdir -p .volumes/memento cp config-sample.yml .volumes/memento/config.yml ``` -**Start everything** +Start everything ```shell script docker-compose up -d ``` -**Open the dashboard to check progress** +Open the dashboard to check progress ``` http://localhost:3000 ``` @@ -161,19 +161,19 @@ http://localhost:3000 #### Running with Docker If you already have a postgres instance & a redis instance set up and still want the simplest way, you can use the docker image from Dockerhub. -**Copy the config and edit it as needed**, making sure Memento is able to connect to postgres, redis and the Ethereum client of choice. +Copy the config and edit it as needed, making sure Memento is able to connect to postgres, redis and the Ethereum client of choice. ```shell script mkdir -p .volumes/memento cp config-sample.yml .volumes/memento/config.yml ``` -**Start memento in Docker** +Start memento in Docker ```shell script docker run --name memento -d -v /path/to/config/folder:/config/ -p 3000:3000 -p 3001:3001 alethio/memento:latest ``` -**Open the dashboard to check progress** +Open the dashboard to check progress ``` http://localhost:3000 ``` diff --git a/docker-compose-single.yml b/docker-compose.yml similarity index 100% rename from docker-compose-single.yml rename to docker-compose.yml From c99326cce8abd4014771fffd05a5c4936daebbec Mon Sep 17 00:00:00 2001 From: Casian Lacatusu Date: Fri, 15 Nov 2019 17:08:21 +0200 Subject: [PATCH 3/4] update reamde --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index fcf57f6..de31c5d 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ Easily check the system status, perform various actions and manage your configur - [Via dashboard](#via-dashboard) - [Via config file / command line arguments](#via-config-file--command-line-arguments) - [Running](#running) - - [Memento with Lite Explorer](#memento-with-lite-explorerhttpsgithubcomalethioethereum-lite-explorer) + - [Memento with Lite Explorer](#memento-with-lite-explorer) - [Standalone via docker compose](#standalone-via-docker-compose) - [Running with Docker](#running-with-docker) - [Building from source](#building-from-source) From 3b99e56e84c906303fa3cd1ea24ae4d2e7ad6df8 Mon Sep 17 00:00:00 2001 From: Casian Lacatusu Date: Fri, 15 Nov 2019 17:10:15 +0200 Subject: [PATCH 4/4] update reamde --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index de31c5d..2b5783f 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,6 @@ Easily check the system status, perform various actions and manage your configur ## Contents - [Memento - Ethereum scraper and indexer](#memento---ethereum-scraper-and-indexer) - - [Contents](#contents) - [Features](#features) - [Getting started](#getting-started) - [Configuration](#configuration)