feat: SpendingKey self derivation, parallel iteration

[dan-da]

obsoletes #278. See it for discussion/background.

Primary changes since #278:

1. Merged SpendingKeyRangeIter into SpendingKeyIter. So there is just one standard iterator.
2. SpendingKeyIter accepts any type of rust Range. (impl RangeBounds)
3. Added SpendingKeyParallelIter. leverages rayon to utilize all cpu cores.
4. Added a serializable SpendingKeyRange type that should be friendlier for non rust languages.
5. /known_keys rpc now returns a SpendingKeyRange instead of a SpendingKeyRangeIter.
6. added a bunch of tests for deriving/iteration.

changelog:

refactor spending-key types so each key can derive its own children
known_keys rpcs now return iterators instead of Vec
DerivationIndex: u64 -> u128.
fix premine_distribution(). separate test vs non-test. make it aware of current network
impl IntoIterator for SpendingKey
GenerationReceivingAddress::derive_from_seed() -> from_seed()
impl rayon parallel iterators
SpendingKeyIter now starts at child 0, instead of parent_key. fix some DoubleEndedIter issues
iteration tests
add test double_ended_range_iterator_meet_middle
add test double_ended_iterator_to_first_elem
add tests double_ended_range_iterator_to_first_elem, range_iterator_to_last_elem, iterator_nth
separate SpendingKeyParallelIter from SpendingKeyIter
add par_iter tests
iterator accepts any impl RangeBounds
tests to validate RangeBounds conversions
known_keys() rpc returns SpendingKeyRange instead of SpendingKeyIter
only perform extra GenerationSpendingKey assertions for debug build. add comments

[aszepieniec]

Premine distribution depends on the network. I can't see the benefit of having a premine distribution that depends on the network. If the issue is having premine allocations for testing purposes, in principle the only one that's necessary is the devnet wallet used in unit tests. Are there other reasons?

Separate Test and Non-Test Premine Distributions. If I read correctly, there are two implementations of premine_distribution, one marked with a test flag and one marked with a not(test) flag. This strikes me as a really bad idea because it defeats the purpose of testing. We want to test the same thing that is going to be deployed in practice, not something else that we hope is similar.

Ensure Integrity of Existing Premine Receiver Keys. I have a concern, already articulated in #300. The context is that we have already received some public receiving addresses from investors, who ran older versions of the software to generate them. The concern is that refactoring how key pairs are managed will invalidate these premine receiver keys.

I realize that previous refactorings may have already invalidated premine receiver keys. Nevertheless, before approving this PR I want to make sure that it is not making the situation worse.

The most methodical way forward is something along these lines:

• checkout relevant commits (alphanet and betanet respectively)
• use neptune-cli generate-wallet, neptune-cli own-receiving-address and neptune-cli export-seed-phrase
• building on HEAD again, add a premine allocation to the generated receiving addresses
• add a unit test that instantiates a wallet for each of the generated seed phrases
• in the test, verify that these wallets come into possession of their respective premine allocations

If addressing this concern methodically is out of scope for this PR, then a) the test case should not hinder its progress, but also b) please find another way to convince me that this PR is not making the situation worse in regards to this concern.

[dan-da]

I haven't had time to address any comments yet because I have been chasing down a unit-test failure that is driving me crazy.

this branch was passing all tests until I rebased it to master prior to pushing this PR.

It now fails one test: models::state::mempool::tests::reorganization_does_not_crash_mempool.

tldr; @aszepieniec I'm going to need your assistance with this.

The error is:

thread 'models::state::mempool::tests::reorganization_does_not_crash_mempool' panicked at src/models/state/mempool.rs:1521:13:                       
Mempool tx must stay confirmable after each new block has been applied`

After a lot of debugging/tracing/bisecting I discovered that if I modify WalletSecret::nth_generation_spending_key() and WalletSecret::nth_spending_key() to use the old derivation logic then the test passes.

After a lot more debugging and messing around with the test case, I still did not determine a cause. I spent days on this, ruling things out, but not finding a smoking gun, or any way to make the test case pass (unmodified) with the new derivation scheme.

Finally this morning I got a bright idea to do something unusual: rebase master on top of my work-in-progress branch.
With the rebase complete I was able to git bisect and find the exact commit in master that does not play well with the new derivation scheme. Unfortunately, I am at a loss to explain why.

The commit is:

$ git bisect good
698a3dd328cc85883c6ffab6349979a6675e3dba is the first bad commit
commit 698a3dd328cc85883c6ffab6349979a6675e3dba
Author: Alan Szepieniec <alan@neptune.cash>
Date:   Mon Dec 16 15:40:30 2024 +0100

    feat(native-currency): Disallow coinbase with negative fee

 .../blockchain/type_scripts/native_currency.rs     | 51 +++++++++++++++++++++-
 1 file changed, 49 insertions(+), 2 deletions(-)

The only code changes in this commit are triton_asm additions inside impl ConsensusProgram for NativeCurrency { fn library_and_code() }.

From context of the modified code, the test case may be generating a coinbase with negative fee somehow, but I don't see how, or why it would be different between the two derivation schemes. The "must stay confirmable" error message provides no clue I can understand.

It is also possible that the asm additions are buggy somehow. Again though I am perplexed why it would work with the old key derivation scheme, but not the new one. Something odd is going on....

[aszepieniec]

I just wanted to say that I have not forgotten about the request for assistance but I just did not get round to it yet. Busy period.

[dan-da]

Premine distribution depends on the network. I can't see the benefit of having a premine distribution that depends on the network. If the issue is having premine allocations for testing purposes, in principle the only one that's necessary is the devnet wallet used in unit tests. Are there other reasons?

Please see #278 (comment) for motivation/rationale. In short, this fixes the total_premine_amount in Block::genesis block(). But also it is simply gross to be including addresses bech32-decoded from different networks, regardless of network. That seems self-evident to me.

Separate Test and Non-Test Premine Distributions. If I read correctly, there are two implementations of premine_distribution, one marked with a test flag and one marked with a not(test) flag. This strikes me as a really bad idea because it defeats the purpose of testing. We want to test the same thing that is going to be deployed in practice, not something else that we hope is similar.

We are not testing what will be deployed in practice unless/until all pre-mine addresses are listed. What we are testing is a set of stand-ins for the real pre-mine list.

More importantly, note that the cfg(test) version of the function contains these lines:

        let mut addrs = vec![
            // chiefly for testing; anyone can access these coins by generating
            // the devnet wallet as above
            (authority_receiving_address, NeptuneCoins::new(20)),
        ];

In my view these should not be included in the real genesis block because:

1. these are unmined coins that will go to whoever first claims them with the devnet wallet private key.
2. this affects the total_premine_amount embedded in genesis block coinbase.
3. this is clearly intended for testing, but was being used in a non-test context.

now, other solutions are possible rather than having two different versions of the function, and feel free to toss out a suggestion, but that is a common/idiomatic way to separate things that are meant for testing from things that are not.

Ensure Integrity of Existing Premine Receiver Keys. I have a concern, already articulated in #300. The context is that we have already received some public receiving addresses from investors, who ran older versions of the software to generate them. The concern is that refactoring how key pairs are managed will invalidate these premine receiver keys.

technically, this does not invalidate any existing priv/pub keypair.

What it will do is make it so that a given wallet seed will generate different keypairs than in the past. Which is functionally equivalent to invalidating existing keys, unless we were to provide a tool for deriving the old-style keys.

It's an interesting consideration. Upon a quick review, it appears we could support the old derivation. It requires storing the master-seed+derivation index in each key, instead of storing only a child seed. So the keys become larger by the size of DerivationIndex, which is presently a u128 (16 bytes).

Personally, I don't think it is worth it. That is a permanent price to pay for a one-time issue. I would advocate to ask premine-recipients to generate new keys once we are fully locked in on the key format (simplest), or to provide a tool for generating the old keys, and a mechanism to import external keys into neptune-core wallet.

I realize that previous refactorings may have already invalidated premine receiver keys. Nevertheless, before approving this PR I want to make sure that it is not making the situation worse.

If they are already invalidated, then this can't make it any worse. invalidated is invalidated. Seems worth checking into.

The most methodical way forward is something along these lines:

• checkout relevant commits (alphanet and betanet respectively)
• use neptune-cli generate-wallet, neptune-cli own-receiving-address and neptune-cli export-seed-phrase
• building on HEAD again, add a premine allocation to the generated receiving addresses
• add a unit test that instantiates a wallet for each of the generated seed phrases
• in the test, verify that these wallets come into possession of their respective premine allocations

A simpler test would just be to generate the 0th and 1st keypair as of alphanet and then verify that derived keys match those values. If the test passes in alphanet, betanet, and master without this PR, then we know this PR invalidate the keys, else if it fails in master they are already invalidated.

If addressing this concern methodically is out of scope for this PR, then a) the test case should not hinder its progress, but also b) please find another way to convince me that this PR is not making the situation worse in regards to this concern.

I will make the above simplified test.

[dan-da]

I just wanted to say that I have not forgotten about the request for assistance but I just did not get round to it yet. Busy period.

thx. good news (I think). I just rebased this PR on top of master and magically the failing test case is passing again. I don't know why, which concerns me, but it seems good. presently waiting for other tests to complete, as new proofs must be generated.

[dan-da]

I just pushed this PR rebased to master and removed the minor changes inside the Arbtirary impls, which are no longer part of normal builds.

After the rebase all tests are passing again (for unknown reason) so I will begin addressing the review comments.

[dan-da]

hmm, CI failed one of the new iterator tests:

---- models::state::wallet::address::key_iter::tests::par_iter_tests::range_iterator_entire_range stdout ----
892
thread 'models::state::wallet::address::key_iter::tests::par_iter_tests::range_iterator_entire_range' panicked at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/collect/mod.rs:100:5:
893
expected 490 total writes, but got 14

This test passes 10/10 attempts for me on local dev machine. I will need to investigate further (later).

[dan-da]

This test passes 10/10 attempts for me on local dev machine. I will need to investigate further (later).

A panic is thrown inside rayon when calling collect() on the parallel iterator.

The test passes if run on a machine with >= 64 cores, or if run with RAYON_NUM_THREADS=n where n >= 64.

This might be a rayon bug that I'm somehow triggering. Opened rayon-rs/rayon#1224

[dan-da]

This might be a rayon bug that I'm somehow triggering.

nope it was my own bug. the 64 thread thing was a weird red herring. fixed now thanks to @cuviper.

[dan-da]

I will make the above simplified test.

Ok, I made a couple tests, and have info to report, and a question/observation.

First, I checked out the alphanet tag and made a couple of tests that:

1. derives spending keys and compare them to hard-coded spending keys derived at alphanet tag.
2. derives receiving addrs, bech32m encode with network=Alpha and compare them to hard-coded addresses derived and encoded the same way at alphanet tag.

I saved these tests in a new branch alphanet-with-generation-derive-test, commit 196a8b0.

Then I updated to present master 953a93c and cherry-picked the above commit. I pushed the result to branch danda/generation-derive-test, commit 426abb8.

What I find is that both tests fail. This means that something has changed about the GenerationSpendingKey format and/or derivation between alphanet and now. I haven't looked into what changed or when.

anyway, these tests can be updated and merged into master once we are fully decided on the derivation method.

---

The context is that we have already received some public receiving addresses from investors, who ran older versions of the software to generate them. The concern is that refactoring how key pairs are managed will invalidate these premine receiver keys.

my question is: when, and in what format did people provide their receiving addresses? I assume they were bech32m encoded, but for which network? At the alphanet tag, Network::Main did not yet exist. (I'm not presently sure when it was added exactly.) So if it was back then, they would've been encoding with ::to_bech32m(Network::Alpha). If that's the case, it already seems problematic with regards to invalidating the keys...

or am I missing something?

[aszepieniec]

my question is: when, and in what format did people provide their receiving addresses? I assume they were bech32m encoded, but for which network? At the alphanet tag, Network::Main did not yet exist. (I'm not presently sure when it was added exactly.) So if it was back then, they would've been encoding with ::to_bech32m(Network::Alpha). If that's the case, it already seems problematic with regards to invalidating the keys...

or am I missing something?

If memory serves, the only effect the network has on the bech32m encoding of an address is the single character infix (or suffix to the human-readable part). And again if memory serves, we decided that this infix should be "t" for testnet and "m" for everything else. So that if you generate an address it will be compatible with mainnet unless you specifically opt out.

[dan-da]

my question is: when, and in what format did people provide their receiving addresses? I assume they were bech32m encoded, but for which network? At the alphanet tag, Network::Main did not yet exist. (I'm not presently sure when it was added exactly.) So if it was back then, they would've been encoding with ::to_bech32m(Network::Alpha). If that's the case, it already seems problematic with regards to invalidating the keys...
or am I missing something?

If memory serves, the only effect the network has on the bech32m encoding of an address is the single character infix (or suffix to the human-readable part). And again if memory serves, we decided that this infix should be "t" for testnet and "m" for everything else. So that if you generate an address it will be compatible with mainnet unless you specifically opt out.

If I'm reading the bech32 crate code correctly, the HRP also affects the checksum. That is something we can test/verify.

if I understand your point correctly it is that we could modify the HRP of the received addresses before hard-coding them into the code. While it may be technically possible, it strikes me as problematic to modify what we were sent by 3rd parties. Seems to me it should match exactly, so there's no cause for questions/concern.

regardless of all that, the test failure I am seeing is not only the prefix. here is an example:

---- models::state::wallet::wallet_tests::generation_key_derivation::verify_derived_generation_addrs stdout ----
index: 0
thread 'models::state::wallet::wallet_tests::generation_key_derivation::verify_derived_generation_addrs' panicked at src/models/state/wallet/mod.rs:1594:17:
assertion `left == right` failed

  left: "nolgam16n494axmrtxsxhftn2sgvn4uggt6tag8skztcfc8a2yrrn5l69n8gk5f8eenhf0pelr0rt5wxk82z46juq9ndpzx4377hv2ngns06x5hcchvtmr8wxtpvvykujq6tszt2w4mhdwmssknyfpjx59f6ywyz9crc2s4md0dksv0ayklk5rx3duz7p2gjtmtlc3sgz07urxljtf77a9yrwn6qy300f2e36z6humz3rehvphpaddj7vd02ucuktw9ux476njx9hn3sv92ay6up50ef4n3zh2a6jvdfcgsp6ed5k2q8e4lfpmc3p9uyrej7scarvkefe6e2muup8tyn5a4fvfsy48e3rcpxncfkz9wk0j5wss98rv3zq50uddhafh77z93ulleysdm839emh8v69y053v4hvpffr3s5x3lxmshkrq0087lyppqaj5xn52fhu6tkjxg2cvlxuarh85c58vnyatuaytcnshux4tk2qwgmu8kk0s6u34xv643aq59yrsnsvu5wskzvkwzjjrhlzjnhs5z5xg8fyvanv5806cpzjea277950vpt2npf9qr7qd96rttst05fdcxk5j0ut4c7qfpa03tq5rrk6n94m0fzu2km9hc4lez47e0v9yeu0xt6jkpjsvwcrwu0gxnuq4j2qjmcnkvafsuteyayhqvtwm3ypqyknr6zx7zcusp9h29970073wzwywp8gx0t3yh4usw23gvlhauctsfye8g3n4tvfg2xuhgrfr86q5z9u9kkec39krmyvzpewah3c36em0zskkns49jl9q4zynj8rymgaethxsrmjmakj0epe42xctu744ktwp9ms7xh2gumzexfhmgkqpypusx5sv7ag03pstlhqpp3s8vptaeqt4p57ejjlxl3v2zqdvsnvvcjv5twrcgv8s5a5hp27v8vqltxmfz9hd5m2l7yf9ux9d6rehz26gz0fuymnmcpru3500y689wq6k45xacwzhvlxwh0gk84090yxmeng8vwjvs35xkhnew7dn5zxky8g63kyfwhnz2m6vjwl655gwv9wgxynjsyga2h98hqz63eej8lpn6g3u4tar4j4d9ul2es2s28sdg9p0d4mawt7gryc69mvzadteyzxadg5h44vcm94gmynn9860fqxp2wgcwrm9df5vy32ylekjdvgmlxq63x2r3su50d25r3w90cl34pmmexhdeqj7g3kqphagvf5yff92am8f83s49c6y5mc972frs4qmmjew4f260e0tpsruqyxcyqm07kn6jmc7ptkfk0ky3hd3cv8xrjarrru69zqyxstn8d4lzjf2qsuvlktmqu037dfvmws8ejksrp66dqvezyndfz4sr6rcqsaexl8dk3kfyks6g8hdnspehnqcqw4pj8675ql4zwwsq74l0q8nu8a2ww38nz9psdf0t9cszvxk4y755pgfverlg4ew2u742sfv9f0ftwvcmu96p54gfd52hcsg6y0nlzmrdm2yqd08dvcdug9nyl27mchu2puhkmxdwylyuf26fdmya3tm6886sjx0v7ms50f0vsuyhsrewemkgykskldlfaqgfncaz0y60y0yw50t9mlshfezyp2sksn7zgpleuhqmqj5g5dh4ytqnt676ufr3g9aquq6dx8qdkqs5t5ptektsqlrpda9l4slycsy7hz5gyn4dzv45008e0fplhwwxwuarsjace2cr8qnzc55e7uwgs32juxfefflsv5942z5y33p2cgjm983npmw02v82jn9ktrmvadhvcraaz9avp3hpx6vd07pcwk76wml26zr2ew6e4uyjv0455uadvrldq4hev7fh3menu7hk9mvgl7yaez8afn5ysa95uvf4gwg4metjx78js4ssdqj4z5rk20ue0tl9d5k3x9cuefjyxzc6uu9mduke8k4wuz8hfj5wqpv35dzhj3je7g7phrcahrd9u24n04r2g5akupq05trs3h2r924rh53we5p6a3cresh73e9jy5ptr34a3fnhxlhg8gwn0uz5ra27lw5j392zhmjype9qtwmgrhm6y7whqqwkukmwthq40t2hd4j8ld2mckv3fyy97wcyf8dzjqnnqmcwvw0l4uwtl6e9z77w3mrenasvtdre320jhzq4phskk3q5r27avt6fa3k0j0evd8sanpgq6wtk0gssa3tlhstev2fuwpcf87h20v3apfvuglqj6kf4ra85x7zwks5g5tklfkxwswjlgheccypj4832wfu76gggxwvm9vzy5sxttns983z7qqul48ndp0u9268gj2l4qvxzv0r2xvzwc3nanc6sazwwgjc5fy6fg5vezsq93yft3znpwjpm4hqzu4mh2he84ru88gq43hrk4xld6et72m23cts8mlv0c9wc0shjg9jt7u5jn9le095jm32nt7dm257j6de0ym06ah3rpljfwf23gyf2ms8g2dj8hvyc59u7aquj35ajqhvjw55vhn8r2gdap607puwzvwlrvts0mrtsnkqnjq5u84pc39pf9x4pxhv0aghxwmj5wkqx9ynlcwh99wggeh2wk20f79anchlhe645lfzed3u82cf76yza8vhaz2lh42umza4hwfpz20mvjw6thm0vnj3etxvdeu73mjemv4mw27dhwku0726446tklc7memzet7ppdj9x3jvrmsstspt23zjnx8dl9w8akw88hyhlhtglal2gejqf9ktnuhng9p6paasth3sc5x7yjhtpvxr2ma435lfr4jceu6pn0na7n5h37qwuahtac9cpxuley7dvy0sslkq4n0nz0dwj9660pwkymmdj5e8mjnk5r2d2v8qgp4ymz83306teu3ge5rgjlmx0wnz5vpx6rtmgfhk2rphntwgmxd4c4m5cxt4q4y2lz03j0chqrrvjqycwe5kyr0tnhg954wntsx2fzgjx9pma7hq9qtfzweawps7j3mrzxkeg8eh9ve247tnyu4lqmx2f5wx2ql4slfkx2vd4a53n95ymt4frmr89jp8fx59c9jqevwzqzl5fz08clkn5wzawac8qhywryldnwvsjt8tz6s7w5qa85sqzyj23ep58lw2rl0ev8hez534cj2qyw34f8mue8g38s75pa2nju478qq6ylatguat4dam0r8vpepmslm3da3t8nrwm7gjwe968u0ps5nksp3d9svfptudr9sacqvxjcspru2vwzk099uq0pt38clvr3ezmuyq5cjg0ajvn70x8s7qfla3j5w8nwgrrssqfskcxk62zd9k6ssv26vm3p2g5n3lnvhd3dpv87l9hv5w0mvm4hl705t7cfm9tuc7ayxz5jux3u33vvlc84p5ewe5ruzcl7h2gy8u0ehqd38jz5335tme458ndm983gmhfqmcg38uukkzv8p9wz40a6e6"
 right: "ngam10y756kkjqlye95arwt9mgfwqgu406xksep6yg8n6p3r6xv8e6kljpemdul4h7440zp5065q0nyw3w8mzt39wjfgt5vy70au4la6yd96n2acq0f6wreuvvjhq7yan74eat6nudjxvzrnxtys00yxpk8yuzprfhzzhnr5pz9mfaj69kyrv4xk5hyzu9rk0dj93zh977wpf9ad7qh9036sn996gkkdxssqz896uwv8h4pc84afaqlpvh025slwt59nyse9nt9lt9p3uh0mr3pc944cgf585yprjnrm05gjqlu0w0jy8l7cr5dd7n6sd7wykf5ztqk2588ayjsqq8wm880quhku0djk3jn6whwds47tjrdmlpjdudwp44efkgtj2glymn8qswutp8l8wj2kpyesnr5ejhlexthq39jypudddx6mp27n8alepy3k9vqqypfjxuwlrx6j54l294cg4yh9wsxmv2ux6jx38ctxn4lmg6lpn4pamrl99z246xzm6c328zw2apn8ljr5pcmhzwz69weukevxtnfr99s9wfqx0fa8ps4uwn8e95m4s68wevp2kvnya8nj00nngfydvhk2fe6ukav8kmsl98dkd5nm2rycrjjmczmvcxts5zlvw9t52lu47s0htwzhy7mr49mkkafwycmv6gxrphuszlm04ksxrvz4zwrkj5w9w5qmy0z5rgp3hc5mja293qvtxhq5taw9jk0cn2cnu5d9fw3nvh5qzzevmdngnuprg234jx6ayycwuw7nng27pzkms0jjqj2wvjd4wp97sudvklqawyyyhnxshx3ydtsy7ztrxxfctkfsj5ug0lf72qvh5yfa07yvg6xnxarnclc8vwl49x3f2gzhm6gjdgwgjtnsuzjhx6unsgccfcw986k2e0fx76wlu2774r8prduflepyeqw9dnzf6795p7fg35r7dz5y4yu6ew4973smmafwnscy9e69h9kvpe3zjpjthke3mtj25ryd0mhy92paqsyzq7yd7xcekp63ryx0xvrdgq06aghdyra0y4kr0v4meynxfhdvxmvkp2mjzm846v8cf84z0d8y5casmay8zv5hlrz990sfsjd2r89uml7ddzf2ruv2m46q8yefj9sdpvw7nc467q62ysvl08w3sj3c6tz7268kx0c680rfffu4kgm5emp252j6pr79aj7f0fn42mxjt0nwu2shl2je6c8lxak82k3y4srnwat9z0rkjgtx5myadn8mlsvzc0ueuh4k8lyz3hgsdmak6wpzevr9sty85e7mut7r7whczrp9uexgy3xwstn7m82yxsgshrg5x5p4vpmafm86a4nc5ayq3ty54jmavx30ey70gmh3mf507ktr8hg9sx0rl809xd9vwxxw784n0h8h4wkkvgu2lznm5qeryts7yx68m2y32qg4j9utp2sk2ajdhzguh6cja4stj5u8rv4f50rgehhf9yqt8qaxfshcp4n9zh373j0yf2ydalftge3rs3e2xj92fnsm70zkll9hp85ghvsrtf4zhqkxlmq3ly4ystplrrawc2zlx2qpdcvhzzr2rgj9stvcw66229pvgvc6fcftynu4p0cqe0emx7f3ff6n0u70ksh08u7a0pwfgtvx0ze09l0932vlgjs0c6les0nvfpqke57w40gsr6xuljts9auqrl3gjmlht2nglwa5lvcndcte53yekhun9nxg50t525ek68838zah23edrzcsuscaw97mx0l4aem2d3833ussl6tytj3xwqx2jek4rcqvy737q5vgxcejvr0nnxqvdhj0gsklly47z4m74kku50l7r6kk6f9rvx7777fj992zusasq58ldap5jzpa7wfp080ksnh3d522sxgnxy9telufjtwsvdf327jqy6knet3kmkensxfgh6gw4cjsjr6cnd8dw7efvgze3emk9gjh55tewt3xe9p5wwg03l0d9694gq5cwpspsyaezua3yvcjs03gk7vd6z4kfc9fahqdqhhc5ee24juuyau9pljwvsjmdl2pu4e4n5gyea0nm88fc8aeeasnv3jy752gavjq7a6pd4phzjhwdn6e47k29665axpwzzvxhl72wnmcya5wll238pdkgrfy6j874teu0thvx7d3r8037lcmqz7ql69e8aff5fzad5hugh07jkuvx4har3ncarcd2w403ntutu0yhq2mr4k6h03pxsugzzp69vvlc2e0x0a8v599zck0h0faj8wtd037t5tfy8n8nadq4vyhfavpnm3pe5m5yu2jdxu8m9l5gv56twk5mve52ylxlafu8cmrwjrwpn2ll6t2q70r08fqr7pfp9gj6w927rn9m05gdcn4kt459hph4sywhe3ta9cfq74wz83rjy65exhnnkc34uqwcelf2hp2gecxzrhzuggaxgjgqf788ha2tgrf5chtz4j8al8px3vmsgccmeqtw75cyd2uymjwyhjnud8cgppxw2uk0dp6cd62lcpwzgzjc74cuv95vwk228e02kp2ahdgywehuautc5v7cq9cwa993fm08s5njfcnqk0j70qhrqtvu0qn7vwxunshxs0cqy28uyj4gme2k6h4yuvye9ds6c987ez6d9eluhmqjfug7cqwwfwafv6ppts9fz3xchq97edc73c2pherhupdanhcphn8h2j5drmtz3h24rvcaqfdjf4upsmqwjxvs94ssxy69655dqj6xf55cwdjsn6eej2c5t4ln9ytxluc8amn5pz3hcr3qwnfwyqdh9jy4hleftkdh3ewyngaj8ugjfumenqahdfzauhqf3puk7kqaffjyxq0erjk4v9g9g5zql520hl4rut6ys7syxyppd0adu7hx3uj3rxclvj79t03uvzwd0js0z0wd02rf87qu7ax2k6ytkulj8a4yu7y4rfpdxmerdckclyg9wnns6566qx339y3t9e4wusqa4yd2ye9lmfeuhgmzyu0n9yjususgl2wsypzhznxrk6darh25qkhy9mc8y3ejdmgsr0p9maf38pzyxzar8r28slp2mfzsh3vxnjfqwqsshysx5kp3ajkj8shfj3lwra7qx3lp3rnjqm06m8u8cyqwc984ej84rpa4a33au88s6lw3w97qs3xnyqc0um6w7qew9rxug2nh9sfyd5cjdzz8576260c9t6kn4tgcy2xtq5rxafhexn29upfncazyv7w48l9dsja44z2sp0de25nng23s39xcz8m7c9jh9gh0pj0u8jjfa047qey0ymh8m7vgfy2fsu3m3rqvmup0rmzfzhpczdmpfqev4sm2nwq9tsudrt7hl703f92njjd7mc70ah20x86zcrh3xakdy4j6a3cpjum2npexjclz9jkg9g0955rptftmezxfskux42ldkeytr05pwt2l4vc"

where: left is key with index 0, derived by code at 426abb8 (branched from present master) and right is key with index 0, derived by code branched from alphanet tag.

note that:

• left begins with nolgam and right begins with ngam (hrp)
• both start with 1 after the hrp.
• everything else is different after the 1.

I have not attempted to track down when/why/how the change occurs. It can be done, but is a bit tedious.

But it would be helpful to know at what point we received addresses (and if prefixes are ngam or nolgam), because if they are nolgam prefixes, there is still a chance they might not have changed since then.

[aszepieniec]

But it would be helpful to know at what point we received addresses (and if prefixes are ngam or nolgam), because if they are nolgam prefixes, there is still a chance they might not have changed since then.

The first address to be received was on 18 January 2024. All have the nolgam prefix.

[dan-da]

The first address to be received was on 18 January 2024. All have the nolgam prefix.

Thx, that's helpful. I found the alphanet-v5 tag at 06 Jan, 2024. So I conclude that addresses the team received were most likely generated with alphanet-v5 binaries.

Given this, I created branch danda/alphanet-v5-with-generation-derive-test based on alphanet-v5 which adds the tests with keys and addrs generated by the alphanet-v5 code. commit 9870170.

I then adapted these same tests, with same keys and addrs to master, passing Network::Main to ::from_bech32m(). This is in 3f4ddd1.

I'm happy to report that the tests still pass. This means that the addresses we received on 18 Jan 2024 are confirmed compatible with present master, including key derivation, provided they were generated with an alphanet-v5 binary.

Given this, I will next attempt to make the self-deriving keys also compatible and report back.

[dan-da]

Closing in favor of #317 which I believe addresses all review comments.