From 24270512bb7805e8123f829d87027bc32e4f8209 Mon Sep 17 00:00:00 2001 From: Jstar101 <106162669+Jstar101@users.noreply.github.com> Date: Fri, 17 Nov 2023 11:50:36 +0000 Subject: [PATCH] Update README.md Signed-off-by: Jstar101 <106162669+Jstar101@users.noreply.github.com> --- README.md | 787 ++++++++++++++++++++++++++---------------------------- 1 file changed, 383 insertions(+), 404 deletions(-) diff --git a/README.md b/README.md index d0b14156..9204df25 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,16 @@ # StakeWise V3 Operator -## Introduction +1. [What is V3 Operator?]() +2. [Configuring V3 Operator]() +3. [V3 Operator commands]() +4. [Optional Extras]() +5. [Contacts]() + + + + + +# What is V3 Operator? StakeWise Operator is a service that StakeWise Vault operators must run. It is responsible for performing the following tasks: @@ -30,490 +40,381 @@ rotation. The operator periodically checks active validators of the Vault and if some exit signatures become outdated, the operator will submit a signature update transaction to the Vault. -### Vault state update (optional) -The oracles periodically submit consensus rewards of all the vaults to the Keeper contract. -By default, every vault pulls these updates on the user interaction with the vault (deposit, withdraw, etc.), but it -also can be done by the vault operator by passing the `--harvest-vault` flag to the `start` command. Harvesting vault -rewards simplifies calls to the vault contracts, e.g., you don't need to sync rewards before calling deposit. -## Usage -## Step 0. Download operator binary -Download and decompress the binary file from [releases page](https://github.com/stakewise/v3-operator/releases). -## Step 1. Generate keystores & deposit data -Operator generates mnemonic, keystores, deposit data for the validators. It also generates hot wallet used to submit -validator registration transactions. -### How can I find the Vault address? -If you are creating a new Vault: -1. Go to [Operate page](https://app.stakewise.io/operate) -2. Connect with your wallet -3. Click on "Create Vault" -4. Process vault setup step by step -5. Once vault is deployed go to its page -You can find the vault address either in the URL bar or in the "Contract address" field by scrolling to the "Details" at -the bottom of the page. +# Configuring V3 Operator + +Ensure your execution and consensus nodes are fully synced and running. Any execution client that supports [ETH Execution API specification](https://ethereum.github.io/execution-apis/api-documentation/), or consensus client that supports [ETH Beacon Node API specification](https://ethereum.github.io/beacon-APIs/#/), can be used, such as: +**Execution** +[Nethermind](https://launchpad.ethereum.org/en/nethermind) (Ethereum, Gnosis), [Besu](https://launchpad.ethereum.org/en/besu) (Ethereum), [Erigon](https://launchpad.ethereum.org/en/erigon) (Ethereum), and [Geth](https://launchpad.ethereum.org/en/geth) (Ethereum). +**Consensus** +[Lighthouse](https://launchpad.ethereum.org/en/lighthouse) (Ethereum, Gnosis), [Nimbus](https://launchpad.ethereum.org/en/nimbus) (Ethereum), [Prysm](https://launchpad.ethereum.org/en/prysm) (Ethereum), and [Teku](https://launchpad.ethereum.org/en/teku) (Ethereum, Gnosis). -### 1. Init vault config -Create the vault config and mnemonic used to derive validator keys. +## 1. Download and install Operator Service +Operator Service can be run via a binary, built using a docker image, deployed on a Kubernetes cluster using the Operator Helm Chart, or built from source. Decide on your preferred method and follow the respective instructions below. +### Binary +Head to the GitHub repository to find the latest version of Operator Service. Identify the binary file specific to your node hardware, download and decompress it. + +You will execute Operator Service commands from within the `V3-operator` folder using the below format (note that the use of flags is optional): ```bash -./operator init +./operator COMMAND --flagA=123 --flagB=xyz ``` -```sh -Enter the network name (mainnet, goerli, holesky) [mainnet]: -Enter your vault address: 0x3320a...68 -Choose your mnemonic language (chinese_simplified, chinese_traditional, czech, english, italian, korean, portuguese, spanish) [english]: -This is your seed phrase. Write it down and store it safely, it is the ONLY way to recover your validator keys. +Head to [Step 2]() to prepare your Operator Service for launch. +### Docker Image +Build the latest Operator Service docker image using the below command: -pumpkin anxiety private salon inquiry .... +```bash +docker build --pull -t stakewiselabs/v3-operator . +``` +or build a specific version using the below: -Press any key when you have written down your mnemonic. +```bash +docker pull europe-west4-docker.pkg.dev/stakewiselabs/public/v3-operator:v0.3.4 +``` -Please type your mnemonic (separated by spaces) to confirm you have written it down +You will execute Operator Service commands using the below format (note the use of flags are optional): +```bash +docker run --restart on-failure:10 \ +-v ~/.stakewise/:/data \ +europe-west4-docker.pkg.dev/stakewiselabs/public/v3-operator:v0.X.X \ +src/main.py COMMAND \ +--flagA=123 \ +--flagB=xyz +``` -: pumpkin anxiety private salon inquiry .... +Head to [Step 2]() to prepare your Operator Service for launch. -done. -Successfully initialized configuration for vault 0x3320a...68 +### Source Files +If you are running your node on ARM, for example, you will need to build Operator Service from source. Head to the GitHub repository to find the latest version of Operator Service, download and decompress the source files. Note, you must have Python 3.10 and Poetry installed to build from source. +Install the packages required to run Operator Service using Poetry: + +```bash +poetry install --only main ``` -#### Options +You will execute Operator Service commands from within the `V3-operator` folder using the below format (note that the use of flags is optional): -- `--network` - The network of your vault. -- `--vault` - The vault address. -- `--language` - The mnemonic language. -- `--no-verify` - Skips mnemonic verification when provided. -- `--data-dir` - Path where the vault data will be placed. Default is ~/.stakewise. +```bash +pythonpath=. poetry run python src/main.py COMMAND --flagA=123 --flagB=xyz +``` -**NB! You must store the generated mnemonic in a secure cold storage. -It will allow you to restore the keys in case the Vault will get corrupted or lost.** +Head to [Step 2]() to prepare your Operator Service for launch. +### Kubernetes (advanced) +A separate guide runs through the set-up of Operator Service via Kubernetes, designed to run large numbers of validators (up to 10,000). Visit the Kubernetes documentation for more information. -### 2. Create keys -Creates deposit data and validator keystores for operator service: + +## 2. Prepare Operator Service + +In order to run Operator Service, you must first create keystores and deposit data file for your Vault's validators, and set up a hot wallet for Operator Service to handle validator registrations. + +Operator Service has in-built functionality to generate all of the above, or you are free to use your preferred methods of generating keystores and deposit data file, such as via Wagyu Keygen, and your preferred tool for generating the hot wallet, such as MetaMask or MyEtherWallet. + +**Note, the deposit data file must be created using the Vault contract as the withdrawal address. You can find the Vault address either via the URL bar of your Vault page or in the "Contract address" field by scrolling to the "Details" section at the bottom of the Vault page.** + +The below steps walk you through this set-up using Operator Service: +### Creating mnemonic +Run the `init` command and follow the steps to set up your mnemonic used to derive validator keys. For example, if running Operator Service from Binary, you would use: ```bash -./operator create-keys +./operator init +``` +```bash +Enter the network name (goerli) [goerli]: +Enter your vault address: 0x3320ad928c20187602a2b2c04eeaa813fa899468 +Choose your mnemonic language (chinese_simplified, chinese_traditional, czech, english, italian, korean, portuguese, spanish) [english]: +This is your seed phrase. Write it down and store it safely, it is the ONLY way to recover your validator keys. +pumpkin anxiety private salon inquiry .... +Press any key when you have written down your mnemonic. +Please type your mnemonic (separated by spaces) to confirm you have written it down +: pumpkin anxiety private salon inquiry .... +done. +Successfully initialized configuration for vault 0x3320ad928c20187602a2b2c04eeaa813fa899468 ``` -```sh +It is possible to pass the following flags alongside `init`, for example `./operator init --network=goerli` when setting Operator Service up for a Vault on Georli network. +- `--network` - Specified network of your vault (mainnet is default). +- `--language` - Specifies the mnemonic language (english is default). +- `--vault` - Passes vault address (null is default). +- `--no-verify` - Skips mnemonic verification process. +- `--data-dir` - Path where the vault data will be placed (~/.stakewise is default). +**Note, you must store the generated mnemonic in a secure cold storage. This mnemonic is used to restore the validator keys in case the Vault will get corrupted or lost.** +### Creating validator keystores +Next, run the `create-keys` command to kickstart the deposit data and validator keystores creation process, making sure you have your newly created mnemonic to hand: +```bash +./operator create-keys +``` +```bash Enter the vault address: 0x3320a...68 Enter the number of the validator keys to generate: 10 Enter the mnemonic for generating the validator keys: pumpkin anxiety private salon inquiry .... -Creating validator keys: [####################################] 10/10 -Generating deposit data JSON [####################################] 10/10 -Exporting validator keystores [####################################] 10/10 - +Creating validator keys: [####################################] 10/10 +Generating deposit data JSON [####################################] 10/10 +Exporting validator keystores [####################################] 10/10 Done. Generated 10 keys for 0x3320a...68 vault. -Keystores saved to /home/user/.stakewise/0x3320a...68/keystores file -Deposit data saved to /home/user/.stakewise/0x3320a...68/keystores/deposit_data.json file +Keystores saved to /home/user/.stakewise/0x3320ad928c20187602a2b2c04eeaa813fa899468/keystores file +Deposit data saved to /home/user/.stakewise/0x3320ad928c20187602a2b2c04eeaa813fa899468/keystores/deposit_data.json file ``` -#### `create-keys` options - +It is possible to pass the following flags alongside `create-keys`: - `--mnemonic` - The mnemonic for generating the validator keys. - `--count` - The number of the validator keys to generate. -- `--vault` - The vault to generate the keystores and deposit data for. +- `--vault` - The Vault address to generate the keystores and deposit data for. - `--per-keystore-password` - Creates separate password file for each keystore. -- `--data-dir` - Path where the vault data is stored. Default is ~/.stakewise. - -**NB! You must upload the deposit data to your vault:** - -1. Go to [Operate page](https://app.stakewise.io/operate) -2. Connect with your wallet -3. Go to your vault page -4. In the upper right corner, click on "Settings", open the "Deposit data" tab -5. Upload generated deposit data file and click "Save" - -### 3. Create wallet - -Creates the encrypted hot wallet from the mnemonic. -The hot wallet is used to submit validator registration transaction. You must send some ETH (DAI for Gnosis) to the -wallet for the gas expenses. The validator registration costs around 0.01 ETH with 30 Gwei gas price. You must keep an -eye on your wallet balance, otherwise validators will stop registering. +- `--data-dir` - Path where the Vault data is stored (~/.stakewise is default). +**Note, remember to upload the newly generated validator keys to your consensus client. The password for your keystores is located in the passport.txt file in the keystores folder.** +### Create hot wallet +Run the `create-wallet` command to create your hot wallet using your mnemonic (note, this mnemonic can be the same as the one used to generate the validator keys, or a new mnemonic if you desire). ```bash ./operator create-wallet ``` - -```sh +```bash Enter the vault address: 0x3320a...68 Enter the mnemonic for generating the wallet: pumpkin anxiety private salon inquiry ... Done. The wallet and password saved to /home/user/.stakewise/0x3320a...68/wallet directory. The wallet address is: 0x239B...e3Cc ``` - -#### `create-wallet` options - +It is possible to pass the following commands alongside `create-wallet`: - `--vault` - The vault to generate the wallet for. - `--mnemonic` - The mnemonic for generating the wallet. -- `--data-dir` - Path where the vault data is stored. Default is ~/.stakewise. - -Or you can use any of the tools available for generating the hot wallet. For example, - -- [Metamask](https://metamask.io/) - 1. [Generate wallet](https://metamask.zendesk.com/hc/en-us/articles/360015289452-How-to-create-an-additional-account-in-your-wallet) - 2. [Export wallet](https://metamask.zendesk.com/hc/en-us/articles/360015289632-How-to-export-an-account-s-private-key) -- [MyEtherWallet Offline](https://help.myetherwallet.com/en/articles/6512619-using-mew-offline-current-mew-version-6) - -## Step 2. Install execution node - -The execution node is used to fetch data from the Vault contract and to submit transactions. Any execution client that -supports [ETH Execution API specification](https://ethereum.github.io/execution-apis/api-documentation/) can be used: - -- [Nethermind](https://launchpad.ethereum.org/en/nethermind) (Ethereum, Gnosis) -- [Besu](https://launchpad.ethereum.org/en/besu) (Ethereum) -- [Erigon](https://launchpad.ethereum.org/en/erigon) (Ethereum) -- [Geth](https://launchpad.ethereum.org/en/geth) (Ethereum) +- `--data-dir` Path where the vault data is stored (~/.stakewise is default). -## Step 3. Install consensus node +**Note, you must send some ETH (DAI for Gnosis) to the wallet for gas expenses. Each validator registration costs around 0.01 ETH with 30 Gwei gas price. You must keep an eye on your wallet balance, otherwise validators will stop registering if the balance falls too low.** -The consensus node is used to fetch consensus fork data required for generating exit signatures. Any consensus client -that -supports [ETH Beacon Node API specification](https://ethereum.github.io/beacon-APIs/#/) can be used: -- [Lighthouse](https://launchpad.ethereum.org/en/lighthouse) (Ethereum, Gnosis) -- [Nimbus](https://launchpad.ethereum.org/en/nimbus) (Ethereum) -- [Prysm](https://launchpad.ethereum.org/en/prysm) (Ethereum) -- [Teku](https://launchpad.ethereum.org/en/teku) (Ethereum, Gnosis) +## 3. Upload deposit data file to Vault -## Step 4. Run operator service +Now you have created your validator keys, deposit data file, and hot wallet, you now need to upload the deposit data file to the Vault. This process connects your node to the Vault. Note, if there is more than one node operator in a Vault, you first need to merge all operator deposit data files into a single file. Details can be found [here](). +Uploading the deposit data file can be achieved either through the StakeWise UI or via Operator Service and can only be done by the [Vault Admin or Keys Manager](https://docs-v3.stakewise.io/protocol-overview-in-depth/vaults#governance-and-management). +#### StakeWise UI +1. Connect with your wallet and head to the Operate page. +2. Select the Vault you want to upload the deposit data file to. +3. In the upper right corner, click on "Settings" and open the "Deposit Data" tab +4. Upload the deposit data file either by dragging and dropping the file, or clicking to choose the file via your file browser. +5. Click Save and a transaction will be created to sign using your wallet. The Vault's deposit data file will be uploaded when the transaction is confirmed on the network. -### Option 1. From binary executable file +Head to [step 4]() to start Operator Service. -See [releases page](https://github.com/stakewise/v3-operator/releases) to download and decompress the corresponding -binary file. Start the binary with the following command: - -```sh -./operator start --vault=0x3320a...68 --consensus-endpoints=https://consensus.com --execution-endpoints=https://execution.com +#### Operator Service +Run the following command and enter the Vault address when prompted: +```bash +./operator get-validators-root ``` +Note, you can pass the below variables with `get-validators-root` if you are not using the default path storage locations: +- `--data-dir` - Path where the vault data is stored (~/.stakewise is default). +- `--deposit-data-file` - Path to the deposit data file (Vault directory is default). +- `--vault` - The vault address. +After running the command and entering your Vault address, you will be presented with the validator deposit data Merkle tree root. Make a note of this value. -Or you can use environment variables. Check [.env.example](.env.example) file for details - -#### Option 2. Use Docker image - -Build Docker image: - -```sh -docker build --pull -t stakewiselabs/v3-operator . +```bash +The validator deposit data Merkle tree root: 0x50437ed72066c1a09ee85978f168ac7c58fbc9cd4beb7962c13e68e7faac26d7 ``` -or pull existing one: +Finally, upload the Merkle tree root to your Vault contract by calling `setValidatorsRoot`. Below shows the steps to do this via Etherscan, but the same can be achieved via CLI if you prefer (using [eth-cli](https://github.com/protofire/eth-cli) and `eth contract:send` for example). Note, the ABI of the contract can be found [here](https://github.com/stakewise/v3-core/blob/main/abi/IVaultValidators.json). +1. Head to your Vault's contract address page on Etherscan in your browser (e.g. replacing 0x000 with your Vault contract address: https://etherscan.io/address/0x000). +2. Select the Contract tab and then Write as Proxy. +3. Connect your wallet to Etherscan (note this must be either the Vault Admin or Keys Manager). +4. Find the `setValidatorsRoot` function and click to reveal the drop-down. +5. Enter your Merkle tree root and click Write. +6. Confirm the transaction in your wallet to finalize the deposit data upload to your Vault. -```sh -docker pull europe-west4-docker.pkg.dev/stakewiselabs/public/v3-operator:v1.0.0 -``` +You are all set! Now it's time to run the Operator Service. -You have to mount keystores and deposit data folders into docker container. -Start the container with the following command: -```sh -docker run --restart on-failure:10 \ - -v ~/.stakewise/:/data \ - europe-west4-docker.pkg.dev/stakewiselabs/public/v3-operator:v1.0.0 \ - src/main.py start \ - --vault=0x3320a...68 \ - --data-dir=/data \ - --consensus-endpoints=https://example.com \ - --execution-endpoints=https://example.com -``` +## 4. Run Operator Service -Docker compose is an option if you prefer a declarative style instead of long one-liners. -Example `docker-compose.yml` included. Adjust it for yourself and run: - -```sh -docker-compose up +You are now ready to run the Operator Service using the `start` command, passing your Vault address and both consensus and execution endpoints as flags. For example, when running from binary: +```bash +./operator start --vault=0x000... --consensus-endpoints=http://localhost:5052 --execution-endpoints=http://localhost:8545 ``` - -#### Option 3. Use Kubernetes helm chart - -You can use [Operator V3 helm chart](https://github.com/stakewise/helm-charts/tree/main/charts/v3-operator) to host -operator in Kubernetes - -#### Option 4. Build from source - -Build requirements: - -- [Python 3.10+](https://www.python.org/downloads/) -- [Poetry](https://python-poetry.org/docs/) - -Install dependencies and start operator: - -```sh -poetry install --only main +or from source: +```bash PYTHONPATH=. poetry run python src/main.py start \ ---vault=0x3320a...68 \ ---consensus-endpoints=https://example.com \ ---execution-endpoints=https://example.com +--vault=0x000... \ +--consensus-endpoints=http://localhost:5052 \ +--execution-endpoints=http://localhost:8545 ``` - -### Environment variables - -Operator service also can be configured via environment variables instead of cli flags. -Copy [.env.example](.env.example) file to `.env` file and fill it with correct values. -Make sure that file paths in .env file represent vault data and client endpoints. You must load environment variables -before running the operator. - -```sh -export $(grep -v '^#' .env | xargs) -./operator start +For docker, you first need to mount the folder containing validator keystores and deposit data file generated in step 2 into the docker container. You then need to also include the `--data-dir` flag alongside the `start` command as per the below: +```bash +docker run --restart on-failure:10 \ +-v ~/.stakewise/:/data \ +europe-west4-docker.pkg.dev/stakewiselabs/public/v3-operator:v0.3.4 \ +src/main.py start \ +--vault=0x3320ad928c20187602a2b2c04eeaa813fa899468 \ +--data-dir=/data \ +--consensus-endpoints=http://localhost:5052 \ +--execution-endpoints=http://localhost:8545 ``` - -## Remote signer - -You may not want the operator service to have direct access to the validator -keys. Validator keystores do not need to be present directly in the operator. -The operator can query a remote signer to get signatures for validator -exit messages. Because the validator exit signatures are split up and -shared among oracles, the validator exit message needs to be signed by -specific shares of the validator private key. - -These key shares therefore need to be present in your remote signer. - -### Remote signer setup - -This command will split up the private keys in the keystores directory -into private key shares. The resulting private key shares are -then imported to the remote signer. Local keystores are removed -as a result of this command since they no longer need to be present. - -Notes: - -- You will need to run this command every time the oracle set - changes, or the threshold needed to recover exit signatures - (`exit_signature_recover_threshold`) changes. -- In order to regenerate key shares, make sure to - adjust the `mnemonic_next_index` value in the vault config.json - to 0, then run the `create-keys` command, generating the full keystores - for all your validators. Next, run the `remote-signer-setup` command - to regenerate and import the new key shares for all your validators - into the remote signer. - You can remove the previously generated private key shares from the - remote signer, they will not be used anymore. This can optionally be - done by the setup command automatically by using the - `--remove-existing-keys` flag. - +Note, if you did not create your mnemonic using Operator Service, you will need to add the following flags to direct Operator Service to the private key of your hot wallet: +- --hot-wallet-file - path to the password-protected *.txt* file containing your hot wallet private key. +- --hot-wallet-password-file - path to a *.txt* file containing the password to open the protected hot wallet private key file. +Alternatively, an environment file can be created to remove the need for using CLI flags when starting Operator Service. More information can be found [here](). +If using docker, a docker compose file is an option if you wish to simply call `docker run` without adding all the flags manually. `docker-compose.yml` is included in the docker build, edit the file according to your setup using the below command: ```bash -./operator remote-signer-setup \ - --vault=0x3320a...68 \ - --remote-signer-url=http://signer:9000 +docker-compose up ``` -``` text -Successfully generated 11 key shares for 1 private key(s)! -Successfully imported 11 key shares into remote signer. -Removed keystores from local filesystem. -Done. Successfully configured operator to use remote signer for 1 public key(s)! -``` +**Congratulations, you should now have Operator Service up and running and ready to trigger validator registrations within your Vault!** -#### `remote-signer-setup` options -- `--vault` - The vault address. -- `--remote-signer-url` - The base URL of the remote signer, e.g. -- `--remove-existing-keys` - Include this flag to remove any keys present in the signer that are not needed by the - operator. - Can be used to remove outdated keyshares from the remote signer when the set of oracles changes, - see note above. -- `--data-dir` - Path where the vault data is stored. Default is ~/.stakewise. -- `--keystores-dir` - The directory with validator keys in the EIP-2335 standard. -- `--execution-endpoints` - Comma separated list of API endpoints for execution nodes. -- `--verbose` - Enable debug mode. Default is false. -### Running the operator -Provide the operator with the URL to your remote signer instance -using the `--remote-signer-url` flag: -```bash -./operator start --remote-signer-url=http://remote-signer:9000 ... -``` -You should see a message similar to this one after starting the operator: -``` text -Using remote signer at http://remote-signer:9000 for 10 public keys -``` -## Hashi Vault -Operator supports loading signing keys from remote [Hashi Vault](https://github.com/hashicorp/vault) -instance, avoiding storage of keystores on the filesystem. This approach is best suited for -node operators who already have most of Stakewise Operator functionality implemented -in their systems, and only need integration for validator registration or pooling support. -Regular users should only employ this functionality on their own risk, if they already -manage a deployment of hashi vault. -Currently there are two commands that support loading signing keys: `start` and `vaidators-exit`, -user must provide hashi vault instance URL, authentication token, and secret path -in K/V engine. Internal structure of the secret must resemble following json: -```json -{ - "pubkey1": "privkey1", - "pubkey2": "privkey2", - ... -} -``` -Note that public and private signing keys must be stored in hex form, with or -without 0x prefix. -After loading keys from hashi vault, operator behaves in the same way as if it -had loaded them from keystores, no additional operations needed to support -the integration. -## `start` options for hashi vault +# V3 Operator commands -Passing following options to `start` command will enable loading validator signing -keys from remote [Hashi Vault](https://github.com/hashicorp/vault). Make sure -keystores directory is empty before running this command, otherwise operator -will prefer local keystores. +Operator Service has many different commands that are not mandatory but might come in handy: -- `--hashi-vault-url` - URL to the remote hashi vault instance -- `--hashi-vault-token` - Token for use when authenticating with hashi vault -- `--hashi-vault-key-path` - Key path in hashi vault K/V engine holding signing secrets +- [Validators voluntary exit]() +- [Update Vault state (Harvest Vault)]() +- [Update Vault deposit data file]() +- [Merge deposit data files from multiple operators]() +- [Recover validator keystores]() +- [Remote Postgres database (advanced)]() -## Misc commands ### Validators voluntary exit -Performs a voluntary exit for active vault validators. - +Enter the below command to kickstart the voluntary exit process of all your node's validators within a specific Vault: ```bash ./operator validators-exit ``` - -```sh +Follow the steps, confirming your consensus node endpoint, Vault address, and the validator indexes to exit. +```bash Enter the comma separated list of API endpoints for consensus nodes: https://example.com -Enter your vault address: 0x3320a...68 +Enter your vault address: 0x3320ad928c20187602a2b2c04eeaa813fa899468 Are you sure you want to exit 3 validators with indexes: 513571, 513572, 513861? [y/N]: y Validators 513571, 513572, 513861 exits successfully initiated ``` - -#### `validators-exit` options - -- `--network` - The network of your vault. -- `--vault` - The vault address. +There are a variety of variables that can be passed alongside `validators-exit` to customize its function, such as if you wish to only exit one validator in a Vault. They are detailed as follows: +- `--network` - The network of your vault (ethereum by default). +- `--vault` - Your vault address. - `--consensus-endpoints` - Comma separated list of API endpoints for consensus nodes. -- `--count` - The number of validators to exit. By default, command will force exit all active vault validators. +- `--count` - The number of validators to exit (all active vault validators by default). - `--data-dir` - Path where the vault data is stored. Default is ~/.stakewise. - `--remote-signer-url` - URL to the remote signer instance. -- `--hashi-vault-url` - URL to the remote hashi vault instance -- `--hashi-vault-token` - Token for use when authenticating with hashi vault -- `--hashi-vault-key-path` - Key path in hashi vault K/V engine holding signing secrets -- `--verbose` - Enable debug mode. Default is false. +- `--hashi-vault-url` - URL to the remote hashi vault instance. +- `--hashi-vault-token` - Token when authenticating with hashi vault. +- `--hashi-vault-key-path` - Key path in hashi vault K/V engine holding signing secrets. +- `--verbose` - Enable debug mode (false by default). + + +### Update Vault state (Harvest Vault) -### Update vault deposit data +Updating the *Vault state* distributes the Vault fee to the Vault fee address and updates each staker's position. If an ERC-20 token was chosen during Vault creation, the Vault specific ERC-20 reprices based on the rewards/penalties since the previous update and the Vault fees are distributed in newly minted ERC-20 tokens. -You can do that from the StakeWise web app by going to the settings on the vault page and uploading the deposit data or -by using the following command: +By default, each *Vault state* gets updated whenever a user interacts with the Vault (deposit, withdraw, etc.), with a 12 hour cooldown. Vault state can also be updated by the Vault operator(s) by passing the `--harvest-vault` flag to the Operator Service `start` command. Harvest occurs every 24 hours and the gas fees are paid by the hot wallet linked to the Operator Service. -1. Generate deposit data validators root for your vault. +Harvesting the Vault rewards simplifies the contract calls to the Vault contract and reduces the gas fees for stakers, for example, the Vault does not need to sync rewards before calling deposit when a user stakes. - ```bash - ./operator get-validators-root - ``` +### Update Vault deposit data file - ```sh - Enter the vault address: 0xeEFFFD4C23D2E8c845870e273861e7d60Df49663 - The validator deposit data Merkle tree root: 0x50437ed72066c1a09ee85978f168ac7c58fbc9cd4beb7962c13e68e7faac26d7 - ``` +Note, uploading a new deposit data file will overwrite the existing file and consequently overwrite previously un-used validator keys. The can be done at any point, but only by the Vault Admin or Keys Manager. +Run the following command and enter the Vault address when prompted: +```bash +./operator get-validators-root +``` +Note, you can pass the below variables with `get-validators-root` if you are not using the default path storage locations: +- `--data-dir` - Path where the vault data is stored (~/.stakewise is default). +- `--deposit-data-file` - Path to the deposit data file (Vault directory is default). +- `--vault` - The vault address. - `get-validators-root` options +After running the command and entering your Vault address, you will be presented with the validator deposit data Merkle tree root. Make a note of this value. - - `--data-dir` - Path where the vault data is stored. Default is ~/.stakewise. - - `--deposit-data-file` - Path to the file with deposit data. Default is deposit data file located in the vault - directory. - - `--vault` - The vault address. +```bash +The validator deposit data Merkle tree root: 0x50437ed72066c1a09ee85978f168ac7c58fbc9cd4beb7962c13e68e7faac26d7 +``` -2. Set deposit data root by calling `setValidatorsRoot` function on your vault. You must pass the Merkle tree root - generated from the previous command. The ABI of the contract can be - found [here](https://github.com/stakewise/v3-core/blob/main/abi/IVaultValidators.json). +Finally, upload the Merkle tree root to your Vault contract by calling `setValidatorsRoot`. Below shows the steps to do this via Etherscan, but the same can be achieved via CLI if you prefer (using [eth-cli](https://github.com/protofire/eth-cli) and `eth contract:send` for example). Note, the ABI of the contract can be found [here](https://github.com/stakewise/v3-core/blob/main/abi/IVaultValidators.json). +1. Head to your Vault's contract address page on Etherscan in your browser (e.g. replacing 0x000 with your Vault contract address: https://etherscan.io/address/0x000). +2. Select the Contract tab and then Write as Proxy. +3. Connect your wallet to Etherscan (note this must be either the Vault Admin or Keys Manager). +4. Find the `setValidatorsRoot` function and click to reveal the drop-down. +5. Enter your Merkle tree root and click Write. +6. Confirm the transaction in your wallet to finalize the deposit data upload to your Vault. -**NB! The function must be called from the keys manager address (vault admin address by default).** +You are all set! Now it's time to run the Operator Service. -### Recover vault data directory and keystores -```bash +### Merge deposit data files from multiple operators + +Use the following command: +`./operator merge-deposit-data` + +### Recover validator keystores + +Note, for security purposes, make sure to protect your mnemonic as it can be used to generate your validator keys. Always verify the network and endpoints before running the below command to recover your validator keys. +``` bash ./operator recover ``` - -```sh +```bash Enter the mnemonic for generating the validator keys: [Your Mnemonic Here] -Enter your vault address: 0x3320a...68 +Enter your vault address: 0x3320ad928c20187602a2b2c04eeaa813fa899468 Enter comma separated list of API endpoints for execution nodes: https://example.com Enter comma separated list of API endpoints for consensus nodes: https://example.com Enter the network name: goerli Found 24 validators, recovering... -Generating keystores [####################################] 100% +Generating keystores [####################################] 100% Keystores for vault {vault} successfully recovered to {keystores_dir} ``` - -#### `recover` options - -- `--data-dir` - Path where the vault data will be placed. Default is ~/.stakewise. +You can run the following variables alongside the `recover` command: +- `--data-dir` - Path where the Vault data will be placed (~/.stakewise is default). - `--per-keystore-password` - Creates separate password file for each keystore. - `--mnemonic` - The mnemonic for generating the validator keys. - `--vault` - The vault address. - `--execution-endpoints` - Comma separated list of API endpoints for execution nodes. - `--consensus-endpoints` - Comma separated list of API endpoints for consensus nodes. -- `--network` - The network of your vault. Default is Goerli. - -> Note: For security purposes, make sure to protect your mnemonic as it can be used to generate your validator keys. -> Always verify the network and endpoints before running the command. - -### Remote Postgres database +- `--network` - The network of your vault (mainnet is default). -This feature is used in conjunction with the [StakeWise Helm charts](https://github.com/stakewise/helm-charts). It -stores encrypted validator keys and shares in the remote database. -The [web3signer helm chart](https://github.com/stakewise/helm-charts/tree/main/charts/web3signer) pulls the private keys -and decrypts -them on -start. The [validator pods](https://github.com/stakewise/helm-charts/tree/main/charts/validators) use the web3signer -service to sign blocks and fetch the public keys they're validating for -from the DB. The [operator chart](https://github.com/stakewise/helm-charts/tree/main/charts/v3-operator) pulls the -config from the DB and uses -web3signer to sign exit messages. +### Remote Postgres database (advanced) -#### 1. Setup Postgres DB +This feature is only used when running via [Kubernetes and web3signer](https://github.com/stakewise/helm-charts) (advanced), storing encrypted validator keys and shares in a remote database. +The [web3signer helm chart](https://github.com/stakewise/helm-charts/tree/main/charts/web3signer) pulls the private keys and decrypts them on start. The [validator pods](https://github.com/stakewise/helm-charts/tree/main/charts/validators) use the web3signer service to sign blocks and fetch the public keys they're validating for from the DB. The [operator chart](https://github.com/stakewise/helm-charts/tree/main/charts/v3-operator) pulls the config from the DB and uses web3signer to sign exit messages. Follow the steps below to get set up: +#### 1) Setup Postgres DB The command creates tables and generates encryption key for the database: - ```bash ./v3-operator remote-db \ - --db-url=postgresql://postgres:postgres@localhost/operator \ - --vault=0x8189aF89A7718C1baB5628399FC0ba50C6949bCc \ - setup +--db-url=postgresql://postgres:postgres@localhost/operator \ +--vault=0x8189aF89A7718C1baB5628399FC0ba50C6949bCc \ +setup Successfully configured remote database. Encryption key: D/6CbpJen3J0ue0tWcd+d4KKHpT4kaSz3IzG5jz5LFI= NB! You must store your encryption in a secure cold storage! ``` -**NB! You must store the generated encryption key in a secure cold storage. You would have to re-do the setup if you -lose it.** - -#### 2. Load keystores to the remote DB +**NB! You must store the generated encryption key in a secure cold storage. You would have to re-do the setup if you lose it.** +#### 2) Load keystores to the remote DB The command loads encrypted keystores and operator config to the remote DB: - ```bash ./v3-operator remote-db \ - --db-url=postgresql://postgres:postgres@localhost/operator \ - --vault=0x8189aF89A7718C1baB5628399FC0ba50C6949bCc \ - upload-keypairs \ - --encrypt-key=D/6CbpJen3J0ue0tWcd+d4KKHpT4kaSz3IzG5jz5LFI= \ - --execution-endpoints=http://localhost:8545 +--db-url=postgresql://postgres:postgres@localhost/operator \ +--vault=0x8189aF89A7718C1baB5628399FC0ba50C6949bCc \ +upload-keypairs \ +--encrypt-key=D/6CbpJen3J0ue0tWcd+d4KKHpT4kaSz3IzG5jz5LFI= \ +--execution-endpoints=http://localhost:8545 Loading keystores from /Users/user/.stakewise/0x8189af89a7718c1bab5628399fc0ba50c6949bcc/keystores... Fetching oracles config... Calculating and encrypting shares for 10000 keystores... @@ -521,121 +422,199 @@ Uploading updates to the remote db... Successfully uploaded keypairs and shares for the 0x8189aF89A7718C1baB5628399FC0ba50C6949bCc vault. ``` -#### 3. Sync keystores to the web3signer - +#### 3) Sync keystores to the web3signer The command syncs encrypted keystores to the web3signer: - ```bash ./v3-operator remote-db \ - --db-url=postgresql://postgres:postgres@localhost/operator \ - --vault=0x8189aF89A7718C1baB5628399FC0ba50C6949bCc \ - --network=mainnet \ - setup-web3signer \ - --encrypt-key=D/6CbpJen3J0ue0tWcd+d4KKHpT4kaSz3IzG5jz5LFI= \ - --output-dir=./web3signer +--db-url=postgresql://postgres:postgres@localhost/operator \ +--vault=0x8189aF89A7718C1baB5628399FC0ba50C6949bCc \ +--network=mainnet \ +setup-web3signer \ +--encrypt-key=D/6CbpJen3J0ue0tWcd+d4KKHpT4kaSz3IzG5jz5LFI= \ +--output-dir=./web3signer Fetching keypairs from the remote db... Decrypting 120000 keystores... Saving 120000 private keys to web3signer... Successfully retrieved web3signer private keys from the database. ``` -#### 3. Sync web3signer configs for the validators - +#### 4) Sync web3signer configs for the validators The command syncs web3signer config for every validator: - ```bash ./v3-operator remote-db \ - --db-url=postgresql://postgres:postgres@localhost/operator \ - --vault=0x8189aF89A7718C1baB5628399FC0ba50C6949bCc \ - --network=mainnet \ - setup-validator \ - --validator-index=0 \ - --total-validators=10 \ - --web3signer-endpoint=http://localhost:9000 \ - --fee-recipient=0xb793c3D2Cec1d0F35fF88BCA7655B88A44669e4B \ - --output-dir=./validator0 +--db-url=postgresql://postgres:postgres@localhost/operator \ +--vault=0x8189aF89A7718C1baB5628399FC0ba50C6949bCc \ +--network=mainnet \ +setup-validator \ +--validator-index=0 \ +--total-validators=10 \ +--web3signer-endpoint=http://localhost:9000 \ +--fee-recipient=0xb793c3D2Cec1d0F35fF88BCA7655B88A44669e4B \ +--output-dir=./validator0 Generated configs with 1000 keys for validator with index 0. Validator definitions for Lighthouse saved to validator0/validator_definitions.yml file. Signer keys for Teku\Prysm saved to validator0/signer_keys.yml file. Proposer config for Teku\Prysm saved to validator0/proposer_config.json file. - Successfully created validator configuration files. ``` -#### 4. Sync configs for the operator - +#### 5) Sync configs for the operator The command syncs web3signer config and deposit data for the operator: - ```bash ./v3-operator remote-db \ - --db-url=postgresql://postgres:postgres@localhost/operator \ - --vault=0x8189aF89A7718C1baB5628399FC0ba50C6949bCc \ - --network=mainnet \ - setup-operator +--db-url=postgresql://postgres:postgres@localhost/operator \ +--vault=0x8189aF89A7718C1baB5628399FC0ba50C6949bCc \ +--network=mainnet \ +setup-operator Operator remote signer configuration saved to /Users/user/.stakewise/0x8189af89a7718c1bab5628399fc0ba50c6949bcc/remote_signer_config.json file. Operator deposit data saved to /Users/user/.stakewise/0x8189af89a7718c1bab5628399fc0ba50c6949bcc/deposit_data.json file. Successfully created operator configuration file. ``` +By default, the config will be created in the vault directory, but you can override it by providing `--output-dir`. + + + + + + + + -By default, the config will be created in the vault directory, but -you can override it by providing `--output-dir`. -## Monitoring Operator with Prometheus -Operator supports monitoring using Prometheus by providing a `/metrics` endpoint that Prometheus can scrape to gather -various metrics. -### Prerequisites -1. Operator application running and accessible. -2. Prometheus server installed and running. -3. Basic knowledge of how to configure Prometheus targets. -4. [Grafana Dashboard](https://grafana.com/grafana/dashboards/19060-v3-operator/) for `v3-operator` installed -Setup Operator for Monitoring: -Operator provides the flexibility to define the host and port for the metrics endpoint via environment variables: -- `ENABLE_METRICS`: This defines whether the metrics endpoint should be enabled or not. By default, it is set - to `false`. -- `METRICS_HOST`: This defines the hostname or IP on which the metrics endpoint will be available. -- `METRICS_PORT`: This defines the port on which the metrics endpoint will be available. -Ensure that these environment variables are set as per your requirements. -For example: +# Optional Extras +- [Environment variables]() +- [Monitoring Operator with Prometheus]() +- [Remote Signer]() +- [Hashi Vault]() + +### Environment variables + +Operator Service can be configured via environment variables instead of CLI flags. Copy [this example file](https://github.com/stakewise/v3-operator/blob/master/.env.example) and save it as `.env`. Run through the file, adjusting as and where necessary based on your node configuration. + +Remember to load the environment variables before running Operator Service, for example: +```bash +export $(grep -v '^#' .env | xargs) +``` + +You can check the environment variables are set and loaded correctly by running `env`. + + +### Monitoring Operator with Prometheus + +Operator Service supports monitoring using Prometheus by providing a `/metrics` endpoint that Prometheus can scrape to gather various metrics. +#### Prerequisites +1. Operator Service up and running. +2. [Prometheus server](https://prometheus.io/) installed and running. +3. Basic knowledge of how to configure Prometheus targets. +4. [Grafana Dashboard](https://grafana.com/grafana/dashboards/19060-v3-operator/) for `v3-operator` installed and running. +#### Setup Operator Service for monitoring +Operator Service provides the flexibility to define the host and port for the metrics endpoint via environment variables. The endpoint is `http://[METRICS_HOST]:[METRICS_PORT]/metrics`, where: +- `ENABLE_METRICS`: This defines whether the metrics endpoint should be enabled or not. By default, it is set to false. +- `METRICS_HOST`: This defines the hostname or IP on which the metrics endpoint will be available. +- `METRICS_PORT`: This defines the port on which the metrics endpoint will be available. + +Ensure that these environment variables are set as per your requirements, for example, http://0.0.0.0:9100/metrics would be set as: ```bash export ENABLE_METRICS=true export METRICS_HOST=0.0.0.0 export METRICS_PORT=9100 ``` +You can also specify these parameters by providing `--enable-metrics`, `--metrics-port` and `--metrics-host` flags to the `start` command of Operator Service. +#### Configure Prometheus +To monitor Operator Service, you will need to configure Prometheus to scrape metrics from the exposed `/metrics` endpoint. +Add the following job configuration in your Prometheus configuration file (`prometheus.yml`): +```bash +scrape_configs: +- job_name: 'operator' +scrape_interval: 30s +static_configs: +- targets: [ ':' ] +``` +Replace `` and `` with the values you've set in Operator Service. This configuration tells Prometheus to scrape metrics from Operator Service every 30 seconds. +#### Configuring Grafana +Head to your Grafana login screen and Login. Move your mouse over the gear icon at the bottom left of the left menu bar and a menu will pop-up — choose Data Sources. Select Prometheus and enter your endpoint. Click Save & Test to finalize the set-up. -You can also specify them by providing `--enable-metrics`, `--metrics-port` and `--metrics-host` flags to the `start` -command. -Now, Operators's metrics will be available at . -Configure Prometheus: +### Remote signer -To monitor Operator, you will need to configure Prometheus to scrape metrics from the exposed `/metrics` endpoint. +This command will split up the private keys in the keystores directory into private key shares. The resulting private key shares are then imported to the remote signer. Local keystores are removed as a result of this command since they no longer need to be present. +Run the below command, adjusting the vault address and remote signer url based on your set-up: +```bash +./operator remote-signer-setup \ +--vault=0x0000 \ +--remote-signer-url=http://signer:9000 +``` +```bash +Successfully generated 11 key shares for 1 private key(s)! +Successfully imported 11 key shares into remote signer. +Removed keystores from local filesystem. +Done. Successfully configured operator to use remote signer for 1 public key(s)! +``` +You can run the following variables alongside the `remote-signer-setup` command: +- `--vault` - The vault address. +- `--remote-signer-url` - The base URL of the remote signer, e.g. http://signer:9000 +- `--remove-existing-keys` - Include this flag to remove any keys present in the signer that are not needed by Operator Service. Can be used to remove outdated keyshares from the remote signer when the set of Oracles changes, see note below. +- `--data-dir` - Path where the vault data is stored (~/.stakewise is default). +- `--keystores-dir` - The directory with validator keys in the EIP-2335 standard. +- `--execution-endpoints` - Comma separated list of API endpoints for execution nodes. +- `--verbose` - Enable debug mode (false is default). -Add the following job configuration in your Prometheus configuration file (`prometheus.yml`): +Note, you will need to run this command every time the Oracle set changes, or the threshold needed to recover exit signatures (`exit_signature_recover_threshold`) changes. +### Running Operator Service when using remote signer +Provide Operator Service with the URL to your remote signer instance using the `--remote-signer-url` flag: +```bash +./operator start --remote-signer-url=http://remote-signer:9000 ... +``` +You should see a message similar to this one after starting the operator: +```bash +Using remote signer at http://remote-signer:9000 for 10 public keys +``` +### Regenerate key shares +In order to regenerate key shares, make sure to adjust the `mnemonic_next_index` value in the vault config.json to 0, then run the `create-keys` command, generating the full keystores for all your validators. +Next, run the `remote-signer-setup` command to regenerate and import the new key shares for all your validators into the remote signer. You can remove the previously generated private key shares from the remote signer, they will not be used anymore. This can optionally be done by the setup command automatically by using the `--remove-existing-keys` flag. -```yaml -scrape_configs: - - job_name: 'operator' - scrape_interval: 30s - static_configs: - - targets: [ ':' ] + + +### Hashi Vault + +Operator Service supports loading signing keys from a remote [Hashi Vault](https://github.com/hashicorp/vault) instance, avoiding storage of keystores on the filesystem. This approach is best suited for node operators who already have most Operator Service functionality implemented in their systems, and only need it for validator registration or pooling support. Regular users should only employ this functionality on their own risk, if they already manage a deployment of hashi vault. +Currently there are two commands that support loading signing keys: `start` and `vaidators-exit`, user must provide hashi vault instance URL, authentication token, and secret path in K/V engine. The internal structure of the secret must resemble the following json: +```bash +{ +"pubkey1": "privkey1", +"pubkey2": "privkey2", +... +} ``` +Note, public and private signing keys must be stored in hex form, with or without 0x prefix. +After loading keys from hashi vault, operator behaves in the same way as if it had loaded them from keystores, no additional operations needed to support the integration. +Passing following options to `start` command will enable loading validator signing keys from remote a [Hashi Vault](https://github.com/hashicorp/vault). Make sure keystores directory is empty before running this command, otherwise operator will use local keystores. +- `--hashi-vault-url` - URL to the remote hashi vault instance +- `--hashi-vault-token` - Token for use when authenticating with hashi vault +- `--hashi-vault-key-path` - Key path in hashi vault K/V engine holding signing secrets + -Replace `` and `` with the values you've set in Operator. -This configuration tells Prometheus to scrape metrics from Operator every 30 seconds. -## Contacts + + + + +# Contacts - Dmitri Tsumak - - Alexander Sysoev - - Evgeny Gusarov - + + +