Optimize wtxid mining #1329

kpp · 2024-10-11T19:11:00Z

Description

Optimize wtxid mining not only with changing nonce, but also with changin commit/reveal outputs.

Linked Issues

Fixes Bitcoin DA: Optimize WTXID mining #1295

TODO:

optimize witness update on output updates
fix "Too many iterations finding nonce"

Testing

We should run-run-run and check whether it affects wtxid mining.

Note

I tested it 5 times locally on a --release build. Avg time to find a nonce is 2sec for a Proof tx.

codecov · 2024-10-13T11:28:09Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.7%. Comparing base (8cd4668) to head (3eeae73).
Report is 2 commits behind head on nightly.

Additional details and impacted files

Files with missing lines	Coverage Δ
...n-da/src/helpers/builders/batch_proof_namespace.rs	`100.0% <ø> (ø)`
...c/helpers/builders/light_client_proof_namespace.rs	`5.5% <ø> (ø)`
crates/bitcoin-da/src/helpers/builders/mod.rs	`97.8% <100.0%> (+1.2%)`	⬆️
crates/bitcoin-da/src/helpers/builders/tests.rs	`98.0% <100.0%> (+<0.1%)`	⬆️
crates/bitcoin-da/src/service.rs	`44.9% <ø> (-0.1%)`	⬇️

... and 3 files with indirect coverage changes

rakanalh · 2024-10-15T10:36:53Z

@kpp is this still WIP? i can also see " fix "Too many iterations finding nonce" unchecked.

kpp · 2024-10-15T13:34:43Z

@rakanalh just fixed it

yaziciahmet

Great work!

crates/bitcoin-da/src/helpers/builders/batch_proof_namespace.rs

eyusufatik

share bench pls

kpp · 2024-10-18T13:49:42Z

nightly:

/tmp/.tmpACRJkJ/batch-prover/stdout.log
408:2024-10-18T13:37:20.524765Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 9.121542174s, value: 84605

/tmp/.tmpuc82nZ/batch-prover/stdout.log
414:2024-10-18T13:39:43.543477Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 5.6802904s, value: 67262
438:2024-10-18T13:40:07.159919Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 6.235873356s, value: 62570

/tmp/.tmpEgBM0J/batch-prover/stdout.log
415:2024-10-18T13:46:49.340342Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 6.089336519s, value: 71908
439:2024-10-18T13:47:08.012739Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 705.458955ms, value: 7019

/tmp/.tmpiEoT41/batch-prover/stdout.log
414:2024-10-18T13:42:01.928722Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 1.687693373s, value: 20084
438:2024-10-18T13:42:22.500815Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 3.192259856s, value: 32106

/tmp/.tmp3pRiZf/batch-prover/stdout.log
420:2024-10-18T13:44:46.021044Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 13.77681856s, value: 123118

This branch:

/tmp/.tmp687EcV/batch-prover/stdout.log
406:2024-10-18T13:58:51.704095Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 612.227265ms, value: 26

/tmp/.tmpGmvZYG/batch-prover/stdout.log
414:2024-10-18T14:01:22.104855Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 1.0130364s, value: 33

/tmp/.tmpwCKQo4/batch-prover/stdout.log
414:2024-10-18T14:03:40.832588Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 2.373307881s, value: 68
438:2024-10-18T14:03:58.650700Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 132.741848ms, value: 18

/tmp/.tmpEH246F/batch-prover/stdout.log
406:2024-10-18T14:05:56.054776Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 170.969836ms, value: 19

kpp · 2024-10-18T14:14:38Z

kpp · 2024-10-18T14:43:30Z

With 200kb data the perf is:

nightly:

/tmp/.tmpf0bSOX/batch-prover/stdout.log
406:2024-10-18T14:47:21.931412Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 8.349841336s, value: 19044, body size: 200000
434:2024-10-18T14:47:53.313736Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 14.47444719s, value: 32837, body size: 200000

/tmp/.tmpWHLbrH/batch-prover/stdout.log
409:2024-10-18T14:49:31.616966Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 40.066260667s, value: 90858, body size: 200000
433:2024-10-18T14:50:01.623680Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 13.999088812s, value: 31923, body size: 200000

this branch:

/tmp/.tmpUPW85P/batch-prover/stdout.log
414:2024-10-18T14:40:41.999842Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 7.16507906s, value: 43, body size: 200000
438:2024-10-18T14:41:03.410609Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 3.517123143s, value: 29, body size: 200000

/tmp/.tmpNC92IV/batch-prover/stdout.log
414:2024-10-18T14:42:54.548360Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 8.104405025s, value: 45, body size: 200000
438:2024-10-18T14:43:13.800160Z  INFO bitcoin_da::helpers::builders::light_client_proof_namespace: nonce zkp took: 1.285528635s, value: 20, body size: 200000

kpp · 2024-10-22T00:47:56Z

REVEAL_OUTPUT_THRESHOLD is in Sats.

0 Sats:

Took: 56.506059914s, nonce = 88080
Took: 44.852773538s, nonce = 62816
Took: 14.684493909s, nonce = 20672
Took: 8.794750451s, nonce = 12265
Took: 34.472236436s, nonce = 48173
Took: 30.169822022s, nonce = 42195
Took: 13.086576512s, nonce = 18331
Took: 41.290324729s, nonce = 58323
Took: 16.047171079s, nonce = 22737
Took: 5.538695532s, nonce = 7892
Took: 16.62416514s, nonce = 23627
Took: 8.311664398s, nonce = 11821
Took: 47.936773158s, nonce = 68155
Took: 35.688329742s, nonce = 51199
Took: 46.747370585s, nonce = 65645
Took: 18.571181163s, nonce = 26484
Took: 41.624496991s, nonce = 59412
Took: 13.489081868s, nonce = 19272
Took: 47.228255928s, nonce = 67392
Took: 28.801268276s, nonce = 41074
Took: 18.700331367s, nonce = 26435

500 Sats:

Took: 6.563243612s, nonce = 117
Took: 891.097733ms, nonce = 29
Took: 5.934024942s, nonce = 111
Took: 913.160614ms, nonce = 30
Took: 5.317432398s, nonce = 101
Took: 45.680564794s, nonce = 740
Took: 4.420809854s, nonce = 83
Took: 26.413274067s, nonce = 426
Took: 1.517463284s, nonce = 38
Took: 5.079046466s, nonce = 95
Took: 1.028744015s, nonce = 32
Took: 7.751548215s, nonce = 138
Took: 8.84039785s, nonce = 155
Took: 21.381920997s, nonce = 350
Took: 29.925082295s, nonce = 487
Took: 7.77035983s, nonce = 138
Took: 3.260827147s, nonce = 67
Took: 5.705973939s, nonce = 105
Took: 1.217298119s, nonce = 35
Took: 721.147643ms, nonce = 27
Took: 169.562087ms, nonce = 18
Took: 6.152516497s, nonce = 113
Took: 31.666885502s, nonce = 523
Took: 7.595605931s, nonce = 139
Took: 415.364693ms, nonce = 22
Took: 18.926566285s, nonce = 319
Took: 1.463479131s, nonce = 39
Took: 3.787065545s, nonce = 76
Took: 806.745313ms, nonce = 28
Took: 4.494562187s, nonce = 88
Took: 25.353433357s, nonce = 420
Took: 9.421562735s, nonce = 167
Took: 364.423555ms, nonce = 21
Took: 18.967140607s, nonce = 315
Took: 4.380504271s, nonce = 85
Took: 17.347548376s, nonce = 291
Took: 3.202450081s, nonce = 67
Took: 58.764090092s, nonce = 940
Took: 7.152759472s, nonce = 127
Took: 21.627832279s, nonce = 345
Took: 2.702625546s, nonce = 58
Took: 8.125989156s, nonce = 143
Took: 6.427917911s, nonce = 115
Took: 2.870895522s, nonce = 61
Took: 5.116199985s, nonce = 98

2000 Sats:

Took: 2.659433892s, nonce = 27
Took: 7.948321508s, nonce = 50
Took: 42.878077241s, nonce = 198
Took: 2.851185418s, nonce = 28
Took: 4.092011591s, nonce = 33
Took: 7.593648607s, nonce = 48
Took: 2.462457296s, nonce = 26
Took: 560.406297ms, nonce = 18
Took: 7.93992809s, nonce = 50
Took: 6.792312501s, nonce = 45
Took: 3.046892605s, nonce = 29
Took: 892.25921ms, nonce = 19
Took: 1.046605649s, nonce = 20
Took: 3.060234011s, nonce = 28
Took: 2.146028378s, nonce = 24
Took: 5.106314464s, nonce = 37
Took: 6.630040204s, nonce = 43
Took: 7.943535052s, nonce = 49
Took: 6.156102652s, nonce = 41
Took: 2.883890489s, nonce = 27
Took: 6.265514477s, nonce = 41
Took: 3.045431155s, nonce = 28
Took: 1.137761092s, nonce = 20
Took: 2.718424364s, nonce = 27
Took: 1.449593229s, nonce = 21
Took: 4.07905164s, nonce = 32
Took: 4.733408388s, nonce = 34
Took: 2.977924377s, nonce = 28
Took: 825.022748ms, nonce = 19
Took: 2.824397097s, nonce = 26
Took: 12.293694521s, nonce = 66
Took: 15.558299441s, nonce = 81
Took: 36.824388879s, nonce = 173
Took: 13.198238864s, nonce = 72
Took: 5.065044624s, nonce = 37
Took: 18.142965026s, nonce = 91
Took: 15.801984992s, nonce = 80
Took: 4.194348747s, nonce = 33
Took: 8.128898514s, nonce = 49
Took: 4.582554499s, nonce = 35
Took: 5.026780131s, nonce = 36
Took: 1.981273551s, nonce = 23
Took: 3.122920822s, nonce = 29
Took: 42.322487661s, nonce = 189
Took: 7.682921464s, nonce = 48
Took: 592.918905ms, nonce = 18
Took: 4.417016262s, nonce = 34
Took: 1.529330281s, nonce = 22
Took: 7.882830682s, nonce = 48
Took: 673.401967ms, nonce = 18
Took: 3.617630999s, nonce = 31
Took: 503.042966ms, nonce = 18
Took: 2.528570014s, nonce = 26
Took: 356.229239ms, nonce = 17
Took: 1.514138078s, nonce = 22
Took: 18.454561824s, nonce = 91
Took: 4.987692302s, nonce = 36
Took: 6.808082798s, nonce = 44
Took: 453.314769ms, nonce = 17
Took: 15.211506623s, nonce = 79
Took: 3.90824848s, nonce = 32
Took: 1.361720621s, nonce = 21
Took: 1.264223762s, nonce = 21
Took: 15.065586669s, nonce = 79
Took: 1.517389268s, nonce = 22

10000 Sats:

Took: 4.041161093s, nonce = 19
Took: 4.689263182s, nonce = 20
Took: 10.202444535s, nonce = 24
Took: 258.92214ms, nonce = 16
Took: 1.805037912s, nonce = 17
Took: 5.961198148s, nonce = 21
Took: 5.616392682s, nonce = 20
Took: 941.08914ms, nonce = 16
Took: 8.986989086s, nonce = 23
Took: 8.702222172s, nonce = 23
Took: 3.976434178s, nonce = 19
Took: 8.246788956s, nonce = 23
Took: 6.529298559s, nonce = 21
Took: 3.00390354s, nonce = 18
Took: 3.421811085s, nonce = 18
Took: 16.916739736s, nonce = 30
Took: 11.385578698s, nonce = 25
Took: 14.742241755s, nonce = 28
Took: 29.956634417s, nonce = 40
Took: 8.720536279s, nonce = 23
Took: 7.326198717s, nonce = 22
Took: 13.210381387s, nonce = 26
Took: 852.867295ms, nonce = 16
Took: 10.414201234s, nonce = 24
Took: 8.530376955s, nonce = 23
Took: 2.055160895s, nonce = 17
Took: 8.179256574s, nonce = 22
Took: 1.823369302s, nonce = 17
Took: 21.326188536s, nonce = 33
Took: 2.16809747s, nonce = 17
Took: 6.106972673s, nonce = 21
Took: 9.080279219s, nonce = 23
Took: 13.70359573s, nonce = 27
Took: 1.066464798s, nonce = 16
Took: 2.145963836s, nonce = 17
Took: 5.916230378s, nonce = 20
Took: 1.282740828s, nonce = 17
Took: 5.565612957s, nonce = 20
Took: 3.029262713s, nonce = 18
Took: 1.276743653s, nonce = 17
Took: 9.025308414s, nonce = 23
Took: 10.775365797s, nonce = 24
Took: 2.272092585s, nonce = 17
Took: 5.35568131s, nonce = 20
Took: 8.049445205s, nonce = 22
Took: 491.040785ms, nonce = 16
Took: 1.811537017s, nonce = 17
Took: 1.867263976s, nonce = 17
Took: 13.891566623s, nonce = 27
Took: 7.779571679s, nonce = 22
Took: 8.204712073s, nonce = 22
Took: 7.63811559s, nonce = 22
Took: 11.2354546s, nonce = 25
Took: 17.009692763s, nonce = 30
Took: 11.946471851s, nonce = 25
Took: 16.672959983s, nonce = 29
Took: 2.427355982s, nonce = 17
Took: 5.793108477s, nonce = 20
Took: 10.65854109s, nonce = 24
Took: 4.905413214s, nonce = 19
Took: 9.234242599s, nonce = 23
Took: 6.707403978s, nonce = 21
Took: 12.774591625s, nonce = 26
Took: 10.220880745s, nonce = 24
Took: 23.535454395s, nonce = 35
Took: 324.834625ms, nonce = 16
Took: 3.301885886s, nonce = 18
Took: 921.38833ms, nonce = 16
Took: 30.867369202s, nonce = 41
Took: 2.736783909s, nonce = 18
Took: 877.521231ms, nonce = 16
Took: 1.807406818s, nonce = 17
Took: 12.186468659s, nonce = 26
Took: 3.066216034s, nonce = 18
Took: 15.014573192s, nonce = 28
Took: 264.927087ms, nonce = 16

50000 Sats:

Took: 5.9548212s, nonce = 16
Took: 7.500892787s, nonce = 17
Took: 10.901505661s, nonce = 17
Took: 5.63607028s, nonce = 16
Took: 2.835737888s, nonce = 16
Took: 21.290137084s, nonce = 19
Took: 302.124719ms, nonce = 16
Took: 3.264785662s, nonce = 16
Took: 13.489742422s, nonce = 18
Took: 1.692589823s, nonce = 16
Took: 13.531168388s, nonce = 18
Took: 4.25191371s, nonce = 16
Took: 2.099839774s, nonce = 16
Took: 8.171766288s, nonce = 17
Took: 8.807626942s, nonce = 17
Took: 5.230939655s, nonce = 16
Took: 16.019734198s, nonce = 18
Took: 2.683358285s, nonce = 16
Took: 890.652209ms, nonce = 16
Took: 3.457276045s, nonce = 16
Took: 2.759853959s, nonce = 16
Took: 9.01479153s, nonce = 17
Took: 1.894142316s, nonce = 16
Took: 8.958812928s, nonce = 17
Took: 1.318501943s, nonce = 16
Took: 477.557295ms, nonce = 16
Took: 8.442421663s, nonce = 17
Took: 14.91204981s, nonce = 18
Took: 3.157335297s, nonce = 16
Took: 4.034828858s, nonce = 16
Took: 2.19066939s, nonce = 16
Took: 1.422919553s, nonce = 16
Took: 62.404691ms, nonce = 16
Took: 14.534975377s, nonce = 18
Took: 1.512762923s, nonce = 16
Took: 2.068124346s, nonce = 16
Took: 2.717548747s, nonce = 16
Took: 7.358662985s, nonce = 17
Took: 194.923255ms, nonce = 16
Took: 7.541578043s, nonce = 17
Took: 1.149382706s, nonce = 16
Took: 5.53830784s, nonce = 16
Took: 1.378246271s, nonce = 16
Took: 7.739231325s, nonce = 17
Took: 1.918571776s, nonce = 16
Took: 6.994217241s, nonce = 17
Took: 14.460559331s, nonce = 18
Took: 6.987731478s, nonce = 17
Took: 2.76631428s, nonce = 16
Took: 26.574877588s, nonce = 20
Took: 21.349677789s, nonce = 19
Took: 2.690227481s, nonce = 16
Took: 837.199183ms, nonce = 16
Took: 18.405199889s, nonce = 19
Took: 5.057713898s, nonce = 16
Took: 6.521040604s, nonce = 17
Took: 5.284358607s, nonce = 16
Took: 30.721420536s, nonce = 20
Took: 3.068636192s, nonce = 16
Took: 3.018550958s, nonce = 16
Took: 368.652141ms, nonce = 16
Took: 9.477418048s, nonce = 17
Took: 5.478675944s, nonce = 16
Took: 583.053253ms, nonce = 16

eyusufatik

resolve conflict and merge pls

jfldde · 2024-10-27T07:47:16Z

What’s the rationale for not keeping the utxos chained as before ?

kpp added 3 commits October 11, 2024 22:50

Move build_witness into a fn

f9c1735

Move build_taproot into a fn

cdcbbd2

Optimize wtxid mining

daeb84a

kpp force-pushed the kpp/optimize_wtxid_mining branch from 3d79d92 to daeb84a Compare October 11, 2024 20:33

kpp marked this pull request as ready for review October 11, 2024 20:35

Fix tests

dcd805a

kpp added 3 commits October 15, 2024 16:51

Minimal change for a commit_tx is 546 sats

f41d32b

Update condition for Too many iterations finding nonce

b4bdb72

Remove debug timings for commit_tx wtxid mining

87e292c

kpp changed the title ~~WIP: optimize wtxid mining~~ Optimize wtxid mining Oct 15, 2024

yaziciahmet approved these changes Oct 16, 2024

View reviewed changes

kpp commented Oct 16, 2024

View reviewed changes

crates/bitcoin-da/src/helpers/builders/batch_proof_namespace.rs Outdated Show resolved Hide resolved

kpp self-assigned this Oct 16, 2024

Merge branch 'nightly' into kpp/optimize_wtxid_mining

7aba8e1

eyusufatik reviewed Oct 17, 2024

View reviewed changes

kpp added the HOLD-MERGE PR is not draft but should not be merged yet label Oct 17, 2024

kpp added 2 commits October 18, 2024 17:50

Merge branch 'nightly' into kpp/optimize_wtxid_mining

8691c7f

Increase REVEAL_OUTPUT_THRESHOLD from 273 to 2000

1a2dbec

eyusufatik removed the HOLD-MERGE PR is not draft but should not be merged yet label Oct 23, 2024

Merge branch 'nightly' into kpp/optimize_wtxid_mining

f70f3bb

eyusufatik approved these changes Oct 25, 2024

View reviewed changes

kpp added 2 commits October 25, 2024 19:21

Merge branch 'nightly' into kpp/optimize_wtxid_mining

cbdfa1d

Skip chaining tx tests

5274256

Merge branch 'nightly' into kpp/optimize_wtxid_mining

3eeae73

eyusufatik merged commit dd67e3b into nightly Oct 28, 2024
13 of 14 checks passed

eyusufatik deleted the kpp/optimize_wtxid_mining branch October 28, 2024 12:03

jfldde mentioned this pull request Oct 28, 2024

Fix tx chaining tests #1383

Merged

jfldde mentioned this pull request Nov 29, 2024

Remove CI_TEST_MODE #1537

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize wtxid mining #1329

Optimize wtxid mining #1329

kpp commented Oct 11, 2024 •

edited

Loading

codecov bot commented Oct 13, 2024 •

edited

Loading

rakanalh commented Oct 15, 2024

kpp commented Oct 15, 2024

yaziciahmet left a comment

eyusufatik left a comment

kpp commented Oct 18, 2024 •

edited

Loading

kpp commented Oct 18, 2024

kpp commented Oct 18, 2024 •

edited

Loading

kpp commented Oct 22, 2024

eyusufatik left a comment

jfldde commented Oct 27, 2024

Optimize wtxid mining #1329

Optimize wtxid mining #1329

Conversation

kpp commented Oct 11, 2024 • edited Loading

Description

Linked Issues

Testing

Note

codecov bot commented Oct 13, 2024 • edited Loading

Codecov Report

rakanalh commented Oct 15, 2024

kpp commented Oct 15, 2024

yaziciahmet left a comment

Choose a reason for hiding this comment

eyusufatik left a comment

Choose a reason for hiding this comment

kpp commented Oct 18, 2024 • edited Loading

kpp commented Oct 18, 2024

kpp commented Oct 18, 2024 • edited Loading

kpp commented Oct 22, 2024

eyusufatik left a comment

Choose a reason for hiding this comment

jfldde commented Oct 27, 2024

kpp commented Oct 11, 2024 •

edited

Loading

codecov bot commented Oct 13, 2024 •

edited

Loading

kpp commented Oct 18, 2024 •

edited

Loading

kpp commented Oct 18, 2024 •

edited

Loading