Skip to content
/ tools Public

🛠️ Swiss army knife used for running Nevermined Data Platform

License

Apache-2.0 license
3 stars 1 fork Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

nevermined-io/tools

BranchesTags

Repository files navigation

banner

Nevermined Tools

Swiss army knife used for running Nevermined Data Platform

Tests

Prerequisites

You need to have the newest versions of:

  • Linux or macOS. Windows is not currently supported. If you are on Windows, we recommend running the tools inside a Linux VM. Another option might be to use the Windows Subsystem for Linux (WSL).
  • Docker
  • Docker Compose
  • If you want to use Azure Storage (and you might not), then you must edit the file node.env to have your Azure credentials.

Get Started

Once you have all the pre-requisites installed, the first thing we need to do is download the software.

$ git clone https://github.com/nevermined-io/tools.git

If you don't have a Git client, you can download the software from the following URL and unpack in the folder you want:

https://github.com/nevermined-io/tools/archive/refs/heads/master.zip

Once Nevermined Tools is downloaded (and unpacked) you can go to the folder with the software and run it without any parameters. The following command will run the main components of a Nevermined deployment:

  • A local Ethereum node using Geth with the Nevermined Smart Contracts deployed on it
  • An instance of the Marketplace API allowing to register assets Metadata
  • An instance of the Node giving access to off-chain data and services
  • An instance of a Faucet that can be used to get some ETH in the local network
  • An instance of a SubGraph node indexing all the events emitted by the Smart Contracts and exposing them via GraphQL
$ cd tools
$ ./start_nevermined.sh

After running the command you should see something like this:

Welcome to Nevermined

After a few minutes you can run the following command to check that all the Contracts were deployed and Nevermined is ready:

$ ./scripts/wait-nevermined.sh 

Using conf dir: /home/aitor/.nevermined

◯ Waiting for contracts to be generated...
✔ Found new contract artifacts.
[...]
✔ Copied new contract artifacts and circuits.
✔ Nevermined is up and runnning !!!.

Cleaning your environment first (optional)

It's overkill, but to be sure that you use exactly the Docker images and volumes you want, you can prune all the Docker things in your system first:

docker system prune --all --volumes

Get Started on Mac

Due to the differences between Mac and Linux networking and the imposed limitations by Docker, it might happen that the traditional Get Started steps won't work on the new M1 Macs.

Additionally in MacOS you will need to export the variable IP to your hosts ip. This ip must be resolved by your host instance and the containers. The IP assigned in your local network should work. If your Mac has only one network interface, you can get run:

export IP=$(ipconfig getifaddr en0)

If you have multiple network interfaces (i.e.: ethernet and wireless), use en0 if you are connected using ethernet/cable, or en1 if you are connected using wireless.

We strongly adivse to try out the first path, but if you happen to have difficulties, follow the next steps:

git clone git@github.com:nevermined-io/tools.git nevermined-tools
cd nevermined-tools

./start_nevermined.sh --latest --no-marketplace

This way, a subset of the Nevermined stack will be run, with the Legacy Marketplace ommitted.

The focal point of the setup is the Marketplace API. To try the APIs out navigate to the Swagger API description at

http://metadata:5000/api/v1/docs/

You can try it out with exploring the exposed endpoints & dummy requests.

Options

The startup script comes with a set of options for customizing various things.

Component Versions

The default versions are always a combination of component versions which are considered stable.

Contracts Marketplace API Node Faucet
v2.1.0 latest latest v0.2.2

You can use the --latest option to pull the most recent Docker images for all components, which are always tagged as latest in Docker. The latest Docker image tag derives from the default main branch of the component's Git repo.

You can override the Docker image tag used for a particular component by setting its associated environment variable before calling start_nevermined.sh:

  • MARKETPLACE_API_VERSION
  • NODE_VERSION
  • KEEPER_VERSION
  • MARKETPLACE_CLIENT_VERSION
  • MARKETPLACE_SERVER_VERSION
  • FAUCET_VERSION

For example:

export NODE_VERSION=v0.3.0
./start_nevermined.sh

will use the default Docker image tags for Metadata API, Nevermined Contracts and Marketplace, but v0.4.3 for the Node.

If you use the --latest option, then the latest Docker images will be used regardless of whether you set any environment variables beforehand.

All Options

Option Description
--latest Pull Docker images tagged with latest.
--no-marketplace Start up without the marketplace Building Block. Helpful when you are developing the marketplace.
--no-metadata Start up without the metadata Building Block.
--no-gateway Start up without the gateway Building Block.
--no-secret-store Start up without the secret-store Building Block.
--no-faucet Start up without the faucet Building Block.
--no-elastic Start up without ElasticSearch.
--no-graph Start up without the graph node for the Nevermined events.
--compute Start up with the Nevermined compute components.
--ldap Start an OpenLdap instance use for keeping the users and groups authentication.
--dashboard Start up with the dashboard for monitoring containers.
--minio Start up with the minio for the Nevermined arts marketplace.
--polygon-localnet Start up with the a polygon local node for the Nevermined events.
--local-ganache-node Runs a local ganache node.
--local-node or --geth Runs a node of the local geth-localnet network. This is the default.
--local-rinkeby-node Runs a local parity node and connects the node to the rinkeby testnet network
--local-integration-node Runs a local parity node and connects the node to the integration network.
--local-staging-node Runs a local parity node and connects the node to the staging network.
--local-production-node Runs a local parity node and connects the node to the production network
--reuse-ganache-database Configures a running ganache node to use a persistent database.
--force-pull Force pulling the latest revision of the used Docker images.
--purge Removes the Docker containers, volumes, artifact folder and networks used by the script.
--exposeip Binds the components to that specific ip. Exemple: ./start_nevermined.sh --exposeip 192.168.0.1
--deattached Starts the Docker containers in deattached mode

Docker Building Blocks

Barge consists of a set of building blocks that can be combined to form a local test environment. By default all building blocks will be started by the start_nevermined.sh script.

Command Line Interface (CLI)

The command line interface allows to interact with your Nevermined environment in an easy way. If you want to use the CLI in your shell we recommend to install in your local environment using the command:

yarn global add @nevermined-io/cli

OR 

npm install -g @nevermined-io/cli

This will allow to interact with Nevermined using the ncli command.

For more information about different options and/or configuration please visit the CLI repository.

Marketplace

By default it will start two containers (client & server). If the Marketplace is running, you can open the Marketplace Frontend application in your browser:

http://localhost:3000

This Building Block can be disabled by setting the --no-marketplace flag.

Hostname External Port Internal URL Local URL Description
marketplace-client 3000 http://marketplace-client:3000 http://localhost:3000 Marketplace Client
marketplace-server 4000 http://marketplace-server:4000 http://locahost:4000 Marketplace Server

Minio

When passing --minio option it will start a minio container

http://localhost:9000

Hostname External Port Internal URL Local URL Description
nevermined-minio 9000 http://nevermined-minio:9000 http://localhost:9000 Minio used by bazaart

The Graph

When passing --graph option it will start a graph-node container to index Nevermined events.

http://localhost:9000

Hostname External Port Internal URL Local URL Description
nevermined-graph 9000 http://nevermined-graph:9000 http://localhost:9000 The Graph used by Nevermined

Marketplace API

The Marketplace API is a RESTful micro-service that exposes common functionalities that allow building Marketplaces or Dapps around digital assets. When passing --marketplace-api option it will start the Marketplace API container. If the API is running, you can open the API Swagger interface in your browser:

http://localhost:3100/api/v1/docs

Hostname External Port Internal URL Local URL Description
marketplace-api 3100 http://marketplace-api:3100 http://locahost:3100 Marketplace API

Node

By default it will start one container. This Building Block can be disabled by setting the --no-node flag.

Hostname External Port Internal URL Local URL Description
node 8030 http://node:8030 http://localhost:8030 Node

Compute API

By default it will start one container. This Building Block can be enabled by setting the --compute flag.

Hostname External Port Internal URL Local URL Description
compute-api 8050 http://compute-api:8050 http://localhost:8050 Compute API

Keeper Node

Controlled by the --local-*-node config switches will start a container keeper-node that uses port 8545 to expose an rpc endpoint to the Ethereum Protocol. You can find a detailed explanation of how to use this in the script options section of this document.

Hostname External Port Internal URL Local URL Description
keeper-node 8545 http://keeper-node:8545 http://localhost:8545 An Ethereum RPC node

This node can be one of the following types (with the default being geth-localnet):

Node Description
ganache Runs a local ganache-cli node that is not persistent by default. The contracts from the desired nevermined-contracts version will be deployed upon launch of this node.
geth-localnet This is the default. Runs a local node of the geth-localnode Network. The contracts from the desired nevermined-contracts version will be deployed upon launch of this node.
rinkeby Runs a local node connected to the Ethereum Rinkeby Testnet.
integration Runs a local node connected to the Integration Network.
staging Runs a local node connected to the Staging Network.
production Runs a local node connected to the Production Network.

Faucet

By default it will start two containers, one for Faucet server and one for its database (ElasticSearch). This Building Block can be disabled by setting the --no-faucet flag.

Hostname External Port Internal URL Local URL Description
faucet 3001 http://faucet:3001 http://localhost:3001 Faucet

By default the Faucet allows requests every 24hrs. To disable the timespan check you can pass FAUCET_TIMESPAN=0 as environment variable before starting the script.

OpenLdap

If the --ldap flag is given an OpenLdap service will be started.

  • User: admin
  • Password: nevermined
Hostname External Port Internal URL Local URL Description
openldap 1389 ldap://openldap:389 ldap://localhost:1389 OpenLdap

Dashboard

This will start a portainer dashboard with the following admin credentials and connects to the local docker host. This Building Block can be enabled by setting the --dashboard flag.

  • User: admin
  • Password: nevermined
Hostname External Port Internal URL Local URL Description
dashboard 9000 http://dashboard:9000 http://localhost:9000 Portainer

Local Network

If you run the ./start_nevermined.sh script with the --local or --geth option (please see Keeper Node section of this document for more details), you will have available a keeper node in the local and private network with the following accounts enabled:

Account Type Password/Key Balance
0x00Bd138aBD70e2F00903268F3Db08f2D25677C9e key node0 1000000000 Ether
0x068Ed00cF0441e4829D9784fCBe7b9e26D4BD8d0 key secret 1000000000 Ether
0xA99D43d86A0758d5632313b8fA3972B6088A21BB key secret 1000000000 Ether
0xe2DD09d719Da89e5a3D0F2549c7E24566e947260 mnemonic info here 1000000000 Ether
0xBE5449a6A97aD46c8558A3356267Ee5D2731ab5e mnemonic info here 1000000000 Ether
0xA78deb2Fa79463945C247991075E2a0e98Ba7A09 mnemonic info here 1000000000 Ether
0x02354A1F160A3fd7ac8b02ee91F04104440B28E7 mnemonic info here 1000000000 Ether
0xe17D2A07EFD5b112F4d675ea2d122ddb145d117B mnemonic info here 1000000000 Ether
0xA32C84D2B44C041F3a56afC07a33f8AC5BF1A071 mnemonic info here 1000000000 Ether
0xFF3fE9eb218EAe9ae1eF9cC6C4db238B770B65CC mnemonic info here 1000000000 Ether
0x529043886F21D9bc1AE0feDb751e34265a246e47 mnemonic info here 1000000000 Ether
0xe08A1dAe983BC701D05E492DB80e0144f8f4b909 mnemonic info here 1000000000 Ether
0xbcE5A3468386C64507D30136685A99cFD5603135 mnemonic info here 1000000000 Ether

Use one of the above accounts to populate PROVIDER_ADDRESS, PROVIDER_PASSWORD and PROVIDER_KEYFILE in start_nevermined.sh. This account will is used in node and events-handler as the provider account which is important for processing the service agreements flow. The PROVIDER_KEYFILE must be placed in the accounts folder and must match the ethereum address from PROVIDER_ADDRESS. The PROVIDER_ADDRESS is also set in marketplace instance so that published assets get assigned the correct provider address.

Compute Stack

To facilitate the deployment in local of the Nevermined compute stack there is a script called scripts/setup_compute_stack.sh. This script will be in charge of:

  • Install Minikube
  • Install Helm
  • Install the Argo Helm chart
  • Configure the namespace and permissions

So if you want to run the compute stack locally, before running the start_nevermined.sh you can run the scripts/setup_compute_stack.sh script.

Local Mnemonic

The accounts from type mnemonic can be access with this seedphrase:

taxi music thumb unique chat sand crew more leg another off lamp

Attribution

This project is based in the Ocean Protocol Barge. It keeps the same Apache v2 License and adds some improvements. See NOTICE file.

License

Copyright 2022 Nevermined AG
This product includes software developed at
BigchainDB GmbH and Ocean Protocol (https://www.oceanprotocol.com/)

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

About

🛠️ Swiss army knife used for running Nevermined Data Platform

Resources

Readme

License

Apache-2.0 license
Activity
Custom properties

Stars

3 stars

Watchers

7 watching

Forks

1 fork
Report repository

Releases 1

v0.2.0 Latest
Nov 9, 2022

Packages

No packages published

Contributors 11

Languages