Port MPT witness generator

mpt-witness-generator/.github/workflows/go.yml

@@ -0,0 +1,33 @@
+# This workflow will build a golang project
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-go
+
+name: Go
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+
+jobs:
+
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+
+    - name: Set up Go
+      uses: actions/setup-go@v4
+      with:
+        go-version: '1.20'
+
+    - name: Format
+      run: go fmt ./...
+
+    - name: Build
+      run: go build -v ./...
+
+    - name: Test
+      env:
+        NO_GETH: true
+      run: go test -v ./...

mpt-witness-generator/.gitignore

@@ -0,0 +1,7 @@
+rust_call/target
+.DS_Store
+.vscode
+mpt-witness-generator
+mpt
+generated_witnesses
+rust_call/proof.json

mpt-witness-generator/README.md

@@ -0,0 +1,124 @@
+# Merkle Patricia Trie witness generator
+
+This project aims to prepare witness generator for Merkle Patricia Trie circuit which is part of
+[zkevm-circuits](https://github.com/appliedzkp/zkevm-circuits).
+
+It is based on [geth](https://github.com/ethereum/go-ethereum).
+It takes `eth_getProof` output and it transforms it into the MPT circuit witness.  
+
+MPT circuit checks that the modification of the trie state happened correctly.
+
+Let us assume there are two proofs (as returned by `eth getProof`):
+
+- A proof that there exists value `val1` at key `key1` for address `addr` in the state trie with root `root1`.
+- A proof that there exists value `val2` at key `key1` for address `addr` in the state trie with root `root2`.
+
+The circuit checks the transition from `val1` to `val2` at `key1` that led to the change
+of trie root from `root1` to `root2`.
+
+For this reason, there are two parallel proofs for each trie modification.
+There is `S` (as `State`) proof which presents the state of the trie
+before the modification. And there is `C` (as `Change`) proof which presents the state
+of the trie after modification. 
+
+An example of `eth_getProof` output is given below:
+
+```
+[248 81 128 128 128 160 32 34 39 131 73 65 47 37 211 142 206 231 172 16 11 203 33 107 30 7 213 226 2 174 55 216 4 117 220 10 186 68 128 128 128 128 128 128 128 160 55 235 85 86 230 197 53 159 28 141 120 87 82 57 4 132 185 12 24 158 142 210 106 188 12 87 179 231 52 16 126 229 128 128 128 128 128]
+[226 160 59 138 106 70 105 186 37 13 38 205 122 69 158 202 157 33 95 131 7 227 58 235 229 3 121 188 90 54 23 236 52 68 1]
+```
+
+The first element of the proof is a branch containing two children. The second element of the proof is
+a storage leaf contained in the branch (from the first proof element).
+
+To simplify the MPT circuit, the `eth_getProof` elements are not directly used as rows.
+Instead, the rows are prepared as:
+
+```
+[1 0 1 0 248 81 0 248 81 0 11 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
+[0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
+[0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
+[0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
+[0 160 32 34 39 131 73 65 47 37 211 142 206 231 172 16 11 203 33 107 30 7 213 226 2 174 55 216 4 117 220 10 186 68 0 160 32 34 39 131 73 65 47 37 211 142 206 231 172 16 11 203 33 107 30 7 213 226 2 174 55 216 4 117 220 10 186 68 1]
+[0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
+[0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
+[0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
+[0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
+[0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
+[0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
+[0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
+[0 160 55 235 85 86 230 197 53 159 28 141 120 87 82 57 4 132 185 12 24 158 142 210 106 188 12 87 179 231 52 16 126 229 0 160 88 197 127 237 244 146 28 57 104 36 96 69 159 84 254 170 28 196 41 183 253 107 213 32 170 141 111 191 30 100 117 55 1]
+[0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
+[0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
+[0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
+[0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
+[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16]
+[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 17]
+[226 160 59 138 106 70 105 186 37 13 38 205 122 69 158 202 157 33 95 131 7 227 58 235 229 3 121 188 90 54 23 236 52 68 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2]
+[1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 13]
+[226 160 59 138 106 70 105 186 37 13 38 205 122 69 158 202 157 33 95 131 7 227 58 235 229 3 121 188 90 54 23 236 52 68 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3]
+[17 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 14]
+[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15]
+[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19]
+```
+
+...
+
+<!--
+What is changed compared to geth:
+ * Some statedb and trie methods are made public to enable accessing internal structures.
+ -->
+
+## Generate witnesses
+
+To generate witnesses for MPT circuit, go into witness folder and execute
+
+```
+go test gen_witness_from_infura_blockchain_test.go prepare_witness.go leaf.go extension_node.go modified_extension_node.go nodes.go test_tools.go branch.go util.go
+```
+
+to generate the tests that use Infura blockchain.
+
+To generate the tests that use a local blockchain you need a local `geth`. You would
+need to run something like:
+```
+geth --dev --http --ipcpath ~/Library/Ethereum/geth.ipc
+
+```
+The local `geth` is used to generate some tests that have a small number of accounts so that
+these accounts appear in the first or second level of the trie. You might need to remove the
+database if you already have some accounts:
+
+```
+geth removedb
+```
+
+And to generate the tests:
+
+```
+go test gen_witness_from_local_blockchain_test.go prepare_witness.go leaf.go extension_node.go modified_extension_node.go nodes.go test_tools.go branch.go util.go
+```
+
+The witness files will appear in generated_witnesses folder.
+
+## Calling from Rust
+
+Build:
+
+```
+go build -buildmode=c-archive -o libmpt.a witness_gen_wrapper.go 
+```
+
+Copy libmpt.a and libmpt.h to rust_call/build:
+
+```
+mv libmpt.* rust_call/build
+```
+
+Note: to avoid the problem described [](https://github.com/golang/go/issues/42459),
+the following has been set in rust_call/.cargo/config:
+
+```
+[build]
+rustflags = ["-C", "link-args=-framework CoreFoundation -framework Security"]
+```

mpt-witness-generator/go.mod

@@ -0,0 +1,10 @@
+module github.com/privacy-scaling-explorations/mpt-witness-generator
+
+go 1.16
+
+require (
+	github.com/VictoriaMetrics/fastcache v1.6.0 // indirect
+	github.com/ethereum/go-ethereum v1.10.8 // indirect
+	github.com/holiman/uint256 v1.2.0 // indirect
+	golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2 // indirect
+)