[Bitcoin/Rust]: Finalize Bitcoin V2 refactoring (#3848)

* [Bitcoin/V2]: Add a `rust/frameworks` directory, move `tw_utxo` there

* [Bitcoin/V2]: Add Bitcoin Legacy address implementation

* [Bitcoin/V2]: Add `StandardBitcoinAddres`

* [Bitcoin/V2]: Add segwit address

* Switch to TW Bitcoin Address implementation in `BitcoinEntry`

* [Bitcoin/V2]: Implement signAsDER and verifyAsDER used in Bitcoin

* [BitcoinV2]: Add standard Unsigned and Signed Transaction declarations

* Add and implement Bitcoin encoding for Signed Transaction

* [BitcoinV2]: Refactor `TWDerivation` to avoid Coin names

* [BitcoinV2]: Refactor preimage hashing

* Delegate transaction preimage hashing to `LegacyPreimage`, `Witness0Preimage` helpers

* [BitcoinV2]: Design a `TransactionSigner` API

* [BitcoinV2]: Design an API for `TransactionPlanner`, `TransactionBuilder`

* add Script check methods

* cargo fmt

* make signing_method optional, try to automatically determine signing method if unavailable

* start expanding verify_signatures

* add schnorr module to tw_keypair, wip

* expand KeyPair impl for schnorr

* expand enums in tw module with Schnorr

* simplify schnorr module structure

* complete SigningKeyTrait and VerifyingKeyTrait impl for KeyPair

* complete verify_signatures mechanism

* adjust some types, complete compile method

* add test interface for signer, wip

* handle Schnorr variant in tw_keypair

* ignore the recovery byte in ECDSA signature

* update comment

* specify the keys explicitly in tests

* continue debugging

* remove UseDefault from SighashBase, use All as default, fix sighash calculation

* formatting

* add Witness type, correctly consensus-encode items

* fix P2WPKH scriptPubkey generation

* add standard OPCodes and add convenience functions for Script

* complete standard Script builders and checkers

* update script submodule structures

* build claim scripts for standardized transaction variants

* fix Script method call

* update comments, use H264 for new_p2pk instead of PublicKey type

* small cleanup to push_slice

* fix typo

* rename Sighash to SighashType

* expand TransactionBuilder

* adjust builder and add test, wip

* update TransactionBuilder, init test, wip

* complete build_legacy_tx test

* assert in build_legacy_tx

* add BitcoinEcdsaSignature type, serialize correctly for claim script

* add SER_SIZE const to BitcoinEcdsaSignature

* rename TransactionSigner to SighashComputer

* add a ClaimBuilder

* rename some types

* add tests for building Segwit transactions

* delete playgrounds

* track transaction builder test

* add fee calculation traits, implement for standard Transaction, Inputs and Outputs

* implement TransactionFee::fee

* implement CompactInteger::serialized_len

* consider Segwith flag and marker when calculating weight/size/fee

* add select_inputs logic to TransactionFee, handle vbyte/size calculation edge case

* expand tests with size, vsize and weight calculation

* add fee() calculation tests

* add input selection tests, wip

* add base_size to TransactionFee trait, implement method

* move input selection logic out of TransactionFee trait and to ComputedTransaction

* add change amount related methods

* continue with change output build pattern, wip

* mvoe utxo input selector to separate module

* complete test for utxo selection with change output generation

* add note slight fee difference

* extra comments

* introduce CompiledSelectionBuilder

* add p2tr condition builder functions

* add builders for input, claim and output P2TR-key-path scripts

* track taproot1_sighash, wip

* extra conditions

* add tests for Taproot inputs, spending conditions and claims. wip

* continue on Taproot sighash calculation, wip

* continuing with Taproot Sighash debugging...

* add mockup of TapsighHash

* extend test, wip

* remove unused fields in UtxoPreiamgeArgs

* implement Hasher::TapSighash variant

* rename TapSighash prefix const

* seperate sighash methods on PreimageInterface into legacy/segwit and taproot

* move spent_* hashing methods to TransactionHasher

* adjust TapSighash workflow to other variants

* move tx_hasher and sighash_ty to a per-input basis

* add const for Annex indicator

* include leaf_hash and separator as part of TapSighash

* construct Taproot script-path spending info and merkle root, wip

* add p2tr_script_path spending script builder

* init asset/ module, move brc20 and ordinals there

* adjust brc20 and ordinal code for new changes, add brc20_transfer input builder

* separate input builder into individual modules

* fix and format

* remove unused DEFAULT_TX_HASHER in utxo_selector

* build brc20 spending input

* build brc20 transfer output

* add tweak support to tw_keypair

* build brc20 transfer output

* add support for non aux rand data when signing for schnorr, complete P2TR key-path test

* add test for BRC20 transfer output, fix issue with payload

* testing brc20 transfer reveal, wip

* finalize brc20 reveal construction with tests

* get rid of dbg!

* add custom script builders

* wipe tw_bitcoin's preimage_hashes_impl, start with utxo builder migration

* handle custom script proto builder

* start migrating output proto to builder matcher, wip

* use SelectionBuilder

* expand migration of tw_bitcoin preimage_hashes_impl

* create dummy claims for preiamge_hashes_impl

* add and handle a UseAll selector

* complete construction of proto outputs

* add init module with proto_input_to_native function

* implement proto_output_to_native

* process change output

* handle custom script_pubkey outputs

* fix signature issue for p2tr_script_path, expand p2wsh impl

* set send_to_address tests to ignore for now

* remove ignore attribute for ordinals and brc20, all related tests pass

* derive P2PKH and P2SH from base58 address

* derive P2WSH and P2WPKH from bech32 address

* move address to native derivation logic to individual function

* init taproot module with TaprootAddress

* implement p2tr_with_coin_and_prefix for TaprootAddress, derive from StandardBitcoinAddress

* fix tests for reference changes

* distinguish between Segwit and Taproot for StandardBitcoinAddress

* derive Taproot program from address, do extra witness program check

* complete builders for P2WPKH and P2WSH derived from address string

* complete legacy P2PKH and P2SH builders derived from address string

* move address derivation logic to address_to_native

* complete proto builders for p2pkh and p2wpkh from_hash variants

* expand output builder for P2SH, both redeem script and hash

* expand P2SH input claiming, wip

* set ignore attribute to p2sh tests for now

* add option to provide sigs to proto_input_to_native, deprecate some legacy code

* construct proto return value for compile method

* implement txid and fix ownership issue with proto return structure

* update tw_bitcoin calls in wallet_core_rs for legacy FFI functions

* update p2tr_script_path test module, use updated tw_utxo crate

* adjust legacy_script tests to tw_utxo changes

* remove unused imports

* remove print/dbg statements

* remove proto interface for tw_utxo

* add proto_input_to_spending_data function

* rename to proto_sighash_to_sig

* sign sighashes and compile final transaction in tw_bitcon entry

* formatting

* add todo to fn txid

* [Bitcoin]: Cleanups

* [Bitcoin]: Fix most of clippy warnings

* [Bitcoin]: Refactor schnorr private and public keys

* [Bitcoin]: Fix BRC20 tests

* [Bitcoin]: Fix txid

* [Bitcoin]: Temporary add `BitcoinV3.proto` - refactored API

* [Bitcoin]: Deprecate tw_bitcoin/src/modules

* [Bitcoin]: Add `TransactionProtobuf` module

* [Bitcoin]: Use `TWError<SigningError>` in tw_utxo

* [Bitcoin]: Add `UtxoProtobuf` builder

* Merge `SigningMethod::TaprootAll` and `SigningMethod::TaprootOnePrevout`
* Temporary remove `OrdinalInscription`

* [Bitcoin]: Add a `SpendingDataConstructor` pattern

* Refactor `UtxoBuilder` to create a corresponding `SpendingDataConstructor`
* Slightly refactor `BitcoinV3.proto` API

* [Bitcoin]: Finalize UTXO selectors

* Add `MaxSelector`, add `SigningInput::max_amount_output`
* Refactor `SelectorBuilder` as `ExactSelector`
* Add `DustFilter` and `DustPolicy`
* Slightly refactor `BitcoinV3.proto` API

* [Bitcoin]: Finalize UTXO selectors, split `SighashComputer`

* Add `MaxSelector`, add `SigningInput::max_amount_output`
* Refactor `SelectorBuilder` as `ExactSelector`
* Add `DustFilter` and `DustPolicy`
* Slightly refactor `BitcoinV3.proto` API
* Split `SighashComputer` into several modules
* Finalize `TxCompiler`
* Finalize `SighashVerifier`

* [Bitcoin]: Add `TxSigner`

* [Bitcoin]: Add `OutputProtobuf` builder

* Fix Segwit address to not allow witness version > 0

* [Bitcoin]: Refactor `TaprootAddress` and `SegwitAddress`

* [Bitcoin]: Add transaction signer

* [Bitcoin]: Add transaction planner

* [Bitcoin]: Add transaction compiler

* [Bitcoin]: Make `TxPlanner` generic

* [Bitcoin]: Some clean ups

* [Bitcoin]: Remove legacy BRC20 API

* [Bitcoin]: Add BRC20 test

* Add default chain info if not provided
* Add default sequence if not provided

* [Bitcoin]: Add P2PKH test

* [Bitcoin]: Fix final fee calculation

* [Bitcoin]: Add p2tr_key_path

* [Bitcoin]: Move unit tests to `tw_any_coin/tests/chains/bitcoin`

* [Bitcoin]: Fix `send_to_address` tests

* [Bitcoin]: Fix broken tests

* [Bitcoin]: Move UTXO selection to `tw_any_coin/tests`

* [Bitcoin]: Add exact selector with and without change test

* [Bitcoin]: Check transaction weight

* Add more exact_selector tests

* [Bitcoin]: Add more selector tests, increase code coverage

* [Bitcoin]: Add p2pkh compile test

* [Bitcoin]: Add mainnet BCH test

* Fix fork_id sighashing

* [Bitcoin]: Add send to p2sh and p2wsh address tests

* Add p2wpkh signing test
* Set V1 default transaction version

* [Bitcoin]: Add OP_RETURN test

* [Bitcoin]: Allow to pass a DER encoded signature to `TransactionCompiler`

* [Bitcoin]: Test Bitcoin addresses

* [Bitcoin]: Add BRC20 compile tests

* [Bitcoin]: Fix transaction weight

* [Bitcoin]: Remove legacy FFI TWBitcoinFeeCalculateFee

* Return an error if `is_it_brc_operation` is true

* [Bitcoin]: Add P2SH output signing test

* [Bitcoin]: Add p2tr_script_path output test

* Reorg p2tr_key_path tests to cover multiple protobuf types

* [Bitcoin]: Fix p2tr custom script

* [Bitcoin]: Fix P2PK

* Extend P2PKH test to cover all protobuf inputs
* Add P2WSH output test
* Add sighash single test

* [Bitcoin]: Fix some TODOs

* [Bitcoin]: Finalize BitcoinV2

* Remove `Bitcoin.SigningInput.is_it_brc_operation` and `Bitcoin.SigningInput.planning_v2`
* TODO adopt C++ BRC20 tests for BitcoinV2

* [Bitcoin]: Fix one of Android tests

* Add Rust `test_bitcoin_sign_brc20_transfer` test

* [Bitcoin]: Uncomment out Android tests

* [Bitcoin]: Fix iOS tests

* Remove NFT data from Android and iOS test suits

* [Bitcoin]: Fix WASM BRC20 tests

* [CI] Trigger CI

* [Bitcoin]: Minor fixes

* [Bitcoin]: Fix Rust WASM tests

* [Bitcoin]: Add real P2TR transaction tests

* [TON]: Add P2TR test in WASM

---------

Co-authored-by: lamafab <42901763+lamafab@users.noreply.github.com>

Author: satoshiotomakan

android/app/src/androidTest/java/com/trustwallet/core/app/blockchains/TestCoinType.kt

@@ -52,17 +52,17 @@ class TestCoinType {
     fun testDerivationPath() {
         var res = CoinType.createFromValue(CoinType.BITCOIN.value()).derivationPath().toString()
         assertEquals(res, "m/84'/0'/0'/0/0")
-        res = CoinType.createFromValue(CoinType.BITCOIN.value()).derivationPathWithDerivation(Derivation.BITCOINLEGACY).toString()
+        res = CoinType.createFromValue(CoinType.BITCOIN.value()).derivationPathWithDerivation(Derivation.LEGACY).toString()
         assertEquals(res, "m/44'/0'/0'/0/0")
-        res = CoinType.createFromValue(CoinType.SOLANA.value()).derivationPathWithDerivation(Derivation.SOLANASOLANA).toString()
+        res = CoinType.createFromValue(CoinType.SOLANA.value()).derivationPathWithDerivation(Derivation.SOLANA).toString()
         assertEquals(res, "m/44'/501'/0'/0'")
     }
 
     @Test
     fun testDeriveAddressFromPublicKeyAndDerivation() {
         val publicKey = PublicKey("0279BE667EF9DCBBAC55A06295CE870B07029BFCDB2DCE28D959F2815B16F81798".toHexByteArray(), PublicKeyType.SECP256K1)
 
-        val address = CoinType.BITCOIN.deriveAddressFromPublicKeyAndDerivation(publicKey, Derivation.BITCOINSEGWIT)
+        val address = CoinType.BITCOIN.deriveAddressFromPublicKeyAndDerivation(publicKey, Derivation.SEGWIT)
         assertEquals(address, "bc1qw508d6qejxtdg4y5r3zarvary0c5xw7kv8f3t4")
     }
 }

android/app/src/androidTest/java/com/trustwallet/core/app/blockchains/bitcoin/TestBitcoinFee.kt

@@ -53,10 +53,10 @@ class TestAnyAddress {
     fun testCreateWithPublicKeyDerivation() {
         val coin = CoinType.BITCOIN
         val pubkey = PublicKey(any_address_test_pubkey.toHexByteArray(), PublicKeyType.SECP256K1)
-        val address1 = AnyAddress(pubkey, coin, Derivation.BITCOINSEGWIT)
+        val address1 = AnyAddress(pubkey, coin, Derivation.SEGWIT)
         assertEquals(address1.description(), any_address_test_address)
 
-        val address2 = AnyAddress(pubkey, coin, Derivation.BITCOINLEGACY)
+        val address2 = AnyAddress(pubkey, coin, Derivation.LEGACY)
         assertEquals(address2.description(), "1JvRfEQFv5q5qy9uTSAezH7kVQf4hqnHXx")
     }
 

android/app/src/androidTest/java/com/trustwallet/core/app/blockchains/bitcoin/TestBitcoinSigning.kt

@@ -113,13 +113,13 @@ class TestHDWallet {
         val coin = CoinType.BITCOIN
         val wallet = HDWallet(words, password)
 
-        val key1 = wallet.getKeyDerivation(coin, Derivation.BITCOINSEGWIT)
+        val key1 = wallet.getKeyDerivation(coin, Derivation.SEGWIT)
         assertEquals(key1.data().toHex(), "0x1901b5994f075af71397f65bd68a9fff8d3025d65f5a2c731cf90f5e259d6aac")
 
-        val key2 = wallet.getKeyDerivation(coin, Derivation.BITCOINLEGACY)
+        val key2 = wallet.getKeyDerivation(coin, Derivation.LEGACY)
         assertEquals(key2.data().toHex(), "0x28071bf4e2b0340db41b807ed8a5514139e5d6427ff9d58dbd22b7ed187103a4")
 
-        val key3 = wallet.getKeyDerivation(coin, Derivation.BITCOINTESTNET)
+        val key3 = wallet.getKeyDerivation(coin, Derivation.TESTNET)
         assertEquals(key3.data().toHex(), "0xca5845e1b43e3adf577b7f110b60596479425695005a594c88f9901c3afe864f")
     }
 
@@ -137,13 +137,13 @@ class TestHDWallet {
         val coin = CoinType.BITCOIN
         val wallet = HDWallet(words, password)
 
-        val address1 = wallet.getAddressDerivation(coin, Derivation.BITCOINSEGWIT)
+        val address1 = wallet.getAddressDerivation(coin, Derivation.SEGWIT)
         assertEquals(address1, "bc1qumwjg8danv2vm29lp5swdux4r60ezptzz7ce85")
 
-        val address2 = wallet.getAddressDerivation(coin, Derivation.BITCOINLEGACY)
+        val address2 = wallet.getAddressDerivation(coin, Derivation.LEGACY)
         assertEquals(address2, "1PeUvjuxyf31aJKX6kCXuaqxhmG78ZUdL1")
 
-        val address3 = wallet.getAddressDerivation(coin, Derivation.BITCOINTESTNET)
+        val address3 = wallet.getAddressDerivation(coin, Derivation.TESTNET)
         assertEquals(address3, "tb1qwgpxgwn33z3ke9s7q65l976pseh4edrzfmyvl0")
     }
 

android/app/src/androidTest/java/com/trustwallet/core/app/utils/TestAnyAddress.kt

@@ -31,9 +31,9 @@ def self.derivation_name(deriv)
   deriv['name'].downcase
 end
 
-def self.derivation_enum_name(deriv, coin)
+def self.derivation_enum_name(deriv)
   return "TWDerivationDefault" if deriv['name'].nil?
-  "TWDerivation" + format_name(coin['name']) + camel_case(deriv['name'])
+  "TWDerivation" + camel_case(deriv['name'])
 end
 
 def self.coin_img(coin)
@@ -53,6 +53,7 @@ coins = JSON.parse(json_string).sort_by { |x| x['coinId'] }
 
 # used in some cases for numbering enum values
 enum_count = 0
+derivations = {}
 
 erbs = [
   {'template' => 'TWDerivation.h.erb', 'folder' => 'include/TrustWalletCore', 'file' => 'TWDerivation.h'},

android/app/src/androidTest/java/com/trustwallet/core/app/utils/TestHDWallet.kt

@@ -51,7 +51,7 @@ const CoinInfo getCoinInfo(TWCoinType coin) {
                 TWCurve<%= format_name(coin['curve']) %>,
                 {
                     <% coin['derivation'].each do |deriv| -%>{
-                        <%= derivation_enum_name(deriv, coin) %>,
+                        <%= derivation_enum_name(deriv) %>,
                         "<%= deriv['path'] %>",
                         "<%= derivation_name(deriv) %>",
                         TWHDVersion<% if deriv['xpub'].nil? -%>None<% else -%><%= format_name(deriv['xpub']) %><% end -%>,

codegen/bin/coins

@@ -18,11 +18,10 @@ enum TWDerivation {
     TWDerivationCustom = 1, // custom, for any coin
 <% enum_count += 1 -%>
 <% coins.each do |coin| -%>
-<% if coin['derivation'].count > 1 -%>
 <% coin['derivation'].each_with_index do |deriv, index| -%>
-<% if index > 0 or !deriv['name'].nil? -%>
-    <%= derivation_enum_name(deriv, coin) %> = <% enum_count += 1 -%><%= enum_count %>,
-<% end -%>
+<% if !deriv['name'].nil? and !derivations.has_key?(deriv['name']) -%>
+<% derivations[deriv['name']] = true -%>
+    <%= derivation_enum_name(deriv) %> = <% enum_count += 1 -%><%= enum_count %>,
 <% end -%>
 <% end -%>
 <% end -%>

codegen/lib/templates/CoinInfoData.cpp.erb

@@ -191,26 +191,6 @@ struct TWBitcoinScript* _Nonnull TWBitcoinScriptBuildPayToWitnessPubkeyHash(TWDa
 TW_EXPORT_STATIC_METHOD
 struct TWBitcoinScript* _Nonnull TWBitcoinScriptBuildPayToWitnessScriptHash(TWData* _Nonnull scriptHash);
 
-/// Builds the Ordinals inscripton for BRC20 transfer.
-///
-/// \param ticker ticker of the brc20
-/// \param amount uint64 transfer amount
-/// \param pubkey Non-null pointer to a pubkey
-/// \note Must be deleted with \TWBitcoinScriptDelete
-/// \return A pointer to the built script
-TW_EXPORT_STATIC_METHOD
-TWData* _Nonnull TWBitcoinScriptBuildBRC20InscribeTransfer(TWString* _Nonnull ticker, TWString* _Nonnull amount, TWData* _Nonnull pubkey);
-
-/// Builds the Ordinals inscripton for NFT construction.
-///
-/// \param mimeType the MIME type of the payload
-/// \param payload the payload to inscribe
-/// \param pubkey Non-null pointer to a pubkey
-/// \note Must be deleted with \TWBitcoinScriptDelete
-/// \return A pointer to the built script
-TW_EXPORT_STATIC_METHOD
-TWData* _Nonnull TWBitcoinScriptBuildOrdinalNftInscription(TWString* _Nonnull mimeType, TWData* _Nonnull payload, TWData* _Nonnull pubkey);
-
 /// Builds a appropriate lock script for the given address..
 ///
 /// \param address Non-null pointer to an address

codegen/lib/templates/TWDerivation.h.erb

@@ -56,6 +56,7 @@
     "blockchain": "Bitcoin",
     "derivation": [
       {
+        "name": "segwit",
         "path": "m/84'/2'/0'/0/0",
         "xpub": "zpub",
         "xprv": "zprv"
@@ -159,9 +160,16 @@
     "blockchain": "Bitcoin",
     "derivation": [
       {
+        "name": "segwit",
         "path": "m/84'/14'/0'/0/0",
         "xpub": "zpub",
         "xprv": "zprv"
+      },
+      {
+        "name": "legacy",
+        "path": "m/44'/14'/0'/0/0",
+        "xpub": "xpub",
+        "xprv": "xprv"
       }
     ],
     "curve": "secp256k1",
@@ -225,6 +233,7 @@
     "blockchain": "Bitcoin",
     "derivation": [
       {
+        "name": "segwit",
         "path": "m/84'/20'/0'/0/0",
         "xpub": "zpub",
         "xprv": "zprv"
@@ -258,9 +267,16 @@
     "blockchain": "Bitcoin",
     "derivation": [
       {
+        "name": "legacy",
         "path": "m/44'/22'/0'/0/0",
         "xpub": "xpub",
         "xprv": "xprv"
+      },
+      {
+        "name": "segwit",
+        "path": "m/84'/22'/0'/0/0",
+        "xpub": "zpub",
+        "xprv": "zprv"
       }
     ],
     "curve": "secp256k1",
@@ -324,9 +340,16 @@
     "blockchain": "Bitcoin",
     "derivation": [
       {
+        "name": "segwit",
         "path": "m/84'/57'/0'/0/0",
         "xpub": "zpub",
         "xprv": "zprv"
+      },
+      {
+        "name": "legacy",
+        "path": "m/44'/57'/0'/0/0",
+        "xpub": "xpub",
+        "xprv": "xprv"
       }
     ],
     "curve": "secp256k1",
@@ -1517,9 +1540,16 @@
     "blockchain": "Bitcoin",
     "derivation": [
       {
+        "name": "segwit",
         "path": "m/84'/156'/0'/0/0",
         "xpub": "zpub",
         "xprv": "zprv"
+      },
+      {
+        "name": "legacy",
+        "path": "m/44'/156'/0'/0/0",
+        "xpub": "xpub",
+        "xprv": "xprv"
       }
     ],
     "curve": "secp256k1",
@@ -2842,9 +2872,16 @@
     "blockchain": "Bitcoin",
     "derivation": [
       {
+        "name": "legacy",
         "path": "m/44'/2301'/0'/0/0",
         "xpub": "xpub",
         "xprv": "xprv"
+      },
+      {
+        "name": "segwit",
+        "path": "m/84'/2301'/0'/0/0",
+        "xpub": "zpub",
+        "xprv": "zprv"
       }
     ],
     "curve": "secp256k1",
@@ -4243,6 +4280,7 @@
     "blockchain": "Bitcoin",
     "derivation": [
       {
+        "name": "segwit",
         "path": "m/44'/105105'/0'/0/0",
         "xpub": "xpub",
         "xprv": "xprv"