fix(UTXO): calc of txfee with change, min relay fee #2316

dimxy · 2025-01-09T14:50:12Z

Fixes for utxo tx fee calculation in UtxoTxBuilder::build() fn:

include change output in txfee
fix min relay tx fee calc (to multiply by tx_size).

UtxoTxBuilder::build() code was refactored to implement fixed txfee calc: made it similar to daemon code (with a loop), also broke it into functions.
Existing tests updated for the fixed txfee.

Fixes issue: #2313 (a test added to validate txfee from the issue).
Should also fix #1567 issue.

@cipig

fix use gas_fee in build(); fix min_relay_tx_fee calc; refactor break build() into functions rename fn get_tx_fee to get_fee_per_kb; fix utxos tests for recalculated tx fee

* dev: feat(tendermint): validators RPC (#2310) chore(CI): validate Cargo lock file (#2309) test(P2P): add test for peer time sync validation (#2304)

fix refactored UtxoTxBuilder::build(): return only txfee (w/o gas fee) with tx as it used to be

mariocynicys · 2025-01-09T17:09:37Z

Is this ready for review?

dimxy · 2025-01-09T17:18:38Z

Is this ready for review?

I am doing final checks and will change the status for ready after that

borngraced · 2025-01-10T01:44:57Z

Thank you for this PR. Covers most of what I was working on here #2083.

I will close mine when this is approved.

dimxy · 2025-01-10T09:13:41Z

Thank you for this PR. Covers most of what I was working on here #2083.

I will close mine when this is approved.

I thought #2083 was about fee priority (although this PR and #2083 definitely have some interception in txfee calc fixes)

onur-ozkan · 2025-01-22T07:53:03Z

mm2src/coins/utxo/utxo_common.rs

            let total_fee = if tx.outputs.len() == outputs_count {
                // take into account the change output
-                data.fee_amount + (dynamic_fee * P2PKH_OUTPUT_LEN) / KILO_BYTE
+                data.fee_amount + actual_tx_fee.get_tx_fee_for_change(None)


Suggested change

data.fee_amount + actual_tx_fee.get_tx_fee_for_change(None)

data.fee_amount + actual_tx_fee.get_tx_fee_for_change(0)

onur-ozkan · 2025-01-22T07:53:31Z

mm2src/coins/utxo/utxo_common.rs

-                    data.fee_amount
-                }
+                // take into account the change output
+                data.fee_amount + fee_per_kb.get_tx_fee_for_change(Some(tx_bytes.len() as u64))


Suggested change

data.fee_amount + fee_per_kb.get_tx_fee_for_change(Some(tx_bytes.len() as u64))

data.fee_amount + fee_per_kb.get_tx_fee_for_change(tx_bytes.len() as u64)

onur-ozkan · 2025-01-22T07:54:15Z

mm2src/coins/utxo.rs

+    }
+
+    /// Return extra tx fee for the change output as p2pkh
+    fn get_tx_fee_for_change(&self, tx_size: Option<u64>) -> u64 {


I see that tx_size can be used as u64 (see the other suggestions)

onur-ozkan · 2025-01-22T08:15:47Z

mm2src/coins/utxo/utxo_common.rs

+        for utxo in self.required_inputs.clone() {
+            self.tx.inputs.push(UnsignedTransactionInput {
+                previous_output: utxo.outpoint,
+                prev_script: utxo.script,
+                sequence: SEQUENCE_FINAL,
+                amount: utxo.value,
+            });
+            total += utxo.value;
+        }
+
+        for utxo in self.available_inputs.clone() {
+            if total >= amount {
+                break;
+            }
+            self.tx.inputs.push(UnsignedTransactionInput {
+                previous_output: utxo.outpoint,
+                prev_script: utxo.script,
+                sequence: SEQUENCE_FINAL,
+                amount: utxo.value,
+            });
+            total += utxo.value;
+        }


This should be slightly cheaper:

Suggested change

for utxo in self.required_inputs.clone() {

self.tx.inputs.push(UnsignedTransactionInput {

previous_output: utxo.outpoint,

prev_script: utxo.script,

sequence: SEQUENCE_FINAL,

amount: utxo.value,

});

total += utxo.value;

}

for utxo in self.available_inputs.clone() {

if total >= amount {

break;

}

self.tx.inputs.push(UnsignedTransactionInput {

previous_output: utxo.outpoint,

prev_script: utxo.script,

sequence: SEQUENCE_FINAL,

amount: utxo.value,

});

total += utxo.value;

}

for utxo in &self.required_inputs {

self.tx.inputs.push(UnsignedTransactionInput {

previous_output: utxo.outpoint,

prev_script: utxo.script.clone(),

sequence: SEQUENCE_FINAL,

amount: utxo.value,

});

total += utxo.value;

}

for utxo in &self.available_inputs {

if total >= amount {

break;

}

self.tx.inputs.push(UnsignedTransactionInput {

previous_output: utxo.outpoint,

prev_script: utxo.script.clone(),

sequence: SEQUENCE_FINAL,

amount: utxo.value,

});

total += utxo.value;

}

onur-ozkan · 2025-01-22T08:18:58Z

mm2src/coins/utxo/utxo_common.rs

+            if total >= amount {
+                break;
+            }


We should be able to add this check above the loop, so we don't start iterating it for no reason if it's already true.

mariocynicys

Thanks! 1st iteration.
went over all the changes but skipped most of my comments on the tx builder stuff since i couldn't fully comprehend it and don't wanna cause confusion.
will need a couple more iters as this part of the code is a lil risky and confusing.

mariocynicys · 2025-01-22T11:12:14Z

mm2src/coins/utxo.rs

@@ -279,11 +279,43 @@ pub enum TxFee {
 pub enum ActualTxFee {
    /// fee amount per Kbyte received from coin RPC
    Dynamic(u64),
-    /// Use specified amount per each 1 kb of transaction and also per each output less than amount.
+    /// Use specified fee amount per each 1 kb of transaction and also per each output less than the fee amount.


could you explain more about this fee scheme. the per each output part isn't really clear (understood it as 1 fee_rate per 1 output, which i doubt is a correct understanding).

mariocynicys · 2025-01-22T11:14:32Z

mm2src/coins/utxo.rs

+impl ActualTxFee {
+    fn get_tx_fee(&self, tx_size: u64) -> u64 {
+        match self {
+            ActualTxFee::Dynamic(fee_per_kb) => (fee_per_kb * tx_size) / KILO_BYTE,


couldn't this wrongfully floor to zero?

mariocynicys · 2025-01-22T11:16:53Z

mm2src/coins/utxo.rs

@@ -279,11 +279,43 @@ pub enum TxFee {
 pub enum ActualTxFee {


since we corrected the confusing namings, we can fix this one as well to TxFeeRate.

mariocynicys · 2025-01-22T15:13:12Z

mm2src/coins/utxo/utxo_common.rs

+        return_err_if_false!(
            !self.available_inputs.is_empty() || !self.tx.inputs.is_empty(),
            GenerateTxError::EmptyUtxoSet {
-                required: self.sum_outputs_value
+                required: self.required_amount()
            }
        );


what if we refactor this macro to make it return_err_if!. i find it a very challenging mind game to compute the value in there and negate it then. esp that the main action is erroring while the passive one is doing nothing.

return_err_if!(self.available.is_empty() && self.tx.is_empty())

mariocynicys · 2025-01-22T15:14:52Z

mm2src/coins/utxo/utxo_common.rs

-                if self.update_fee_and_check_completeness(from.addr_format(), &actual_tx_fee) {
-                    break;
-                }
+        let mut unused_change;


looks very unusual. let's init the value?

mariocynicys · 2025-01-22T16:24:26Z

mm2src/coins/utxo/utxo_common.rs

+            if self.sum_inputs >= self.sum_outputs + self.total_tx_fee() {
+                break;
+            }


shouldn't this be exactly equal (or equal + unused change)?

mariocynicys · 2025-01-22T16:25:36Z

mm2src/coins/utxo/utxo_common.rs

        let data = AdditionalTxData {
-            fee_amount: self.tx_fee,
-            received_by_me,
+            fee_amount: self.tx_fee, // we return only txfee here (w/o gas_fee)
+            received_by_me: self.sum_received_by_me(&change_script_pubkey),
            spent_by_me: self.sum_inputs,
            unused_change,
            // will be changed if the ticker is KMD


Q: why don't we add this unused/dust change to the fee amount? i mean it's part of the fee at the end of the day.

mariocynicys · 2025-01-22T16:40:16Z

mm2src/coins/utxo/utxo_common.rs

+        let mut one_time_fee_update = false;
+        loop {
+            let required_amount_0 = self.required_amount();


im not really sure about the convergence/termination of this loop. it's not trivial to see.
we can do a 10 rounds or something using a for loop and give up.
and maybe use this for counter to simulate one_time_fee_update.

borngraced

Really great work! Some minor notes from my side

borngraced · 2025-01-22T04:20:12Z

mm2src/coins/utxo/utxo_common.rs

+    fn validate_outputs(&self) -> MmResult<(), GenerateTxError> {
+        for output in self.outputs.iter() {
+            let script: Script = output.script_pubkey.clone().into();
+            if script.opcodes().next() != Some(Ok(Opcode::OP_RETURN)) {
+                return_err_if_false!(output.value >= self.dust(), GenerateTxError::OutputValueLessThanDust {
+                    value: output.value,
+                    dust: self.dust()
+                });
+            }
+        }
+        Ok(())


do you really mean to return Ok if there's no output to validate?

borngraced · 2025-01-23T02:44:39Z

mm2src/coins/utxo/utxo_common.rs

-            change
-        };
+
+            self.sum_outputs -= self.subtract_outputs_by_txfee()?;


shouldn't we use checked_sub here since we didn't validate possible overflowing.
https://doc.rust-lang.org/src/core/num/mod.rs.html#1223-1241

borngraced · 2025-01-23T03:04:52Z

mm2src/coins/utxo/utxo_common.rs

+    let mut tx_fee = fee_per_kb.get_tx_fee(tx_size);
    if coin.as_ref().conf.force_min_relay_fee {
-        let relay_fee = coin.as_ref().rpc_client.get_relay_fee().compat().await?;
-        let relay_fee_sat = sat_from_big_decimal(&relay_fee, coin.as_ref().decimals)?;
-        if fee < relay_fee_sat {
-            fee = relay_fee_sat;
+        let min_relay_fee_per_kb = coin.as_ref().rpc_client.get_relay_fee().compat().await?;
+        let min_relay_fee_per_kb = sat_from_big_decimal(&min_relay_fee_per_kb, coin.as_ref().decimals)?;
+        let min_relay_dynamic_fee = ActualTxFee::Dynamic(min_relay_fee_per_kb);
+        let min_relay_tx_fee = min_relay_dynamic_fee.get_tx_fee(tx_size);
+        if tx_fee < min_relay_tx_fee {
+            tx_fee = min_relay_tx_fee;
        }
    }
-    Ok(fee)
+    Ok(tx_fee)


We can avoid mutation.

let tx_fee = fee_per_kb.get_tx_fee(tx_size); if coin.as_ref().conf.force_min_relay_fee { let min_relay_fee_per_kb = coin.as_ref().rpc_client.get_relay_fee().compat().await?; let min_relay_fee_per_kb = sat_from_big_decimal(&min_relay_fee_per_kb, coin.as_ref().decimals)?; let min_relay_dynamic_fee = ActualTxFee::Dynamic(min_relay_fee_per_kb); let min_relay_tx_fee = min_relay_dynamic_fee.get_tx_fee(tx_size); if tx_fee < min_relay_tx_fee { return Ok(min_relay_tx_fee); } } Ok(tx_fee)

borngraced · 2025-01-23T03:22:24Z

mm2src/coins/utxo/utxo_common.rs

+        if self.sum_inputs > self.sum_outputs + self.total_tx_fee() {
+            let change = self.sum_inputs - (self.sum_outputs + self.total_tx_fee());
+            if change > self.dust() {
+                self.tx.outputs.push({
+                    TransactionOutput {
+                        value: change,
+                        script_pubkey: change_script_pubkey.clone(),
                    }
-                    true
-                } else {
-                    false
-                }
+                });
+                (change, 0u64)
+            } else {
+                (0u64, change)
+            }
+        } else {
+            (0u64, 0u64)
+        }
+    }


what about ?

fn add_change(&mut self, change_script_pubkey: &Bytes) -> (u64, u64) { let sum_output_with_fee = self.sum_outputs + self.total_tx_fee(); if self.sum_inputs < sum_output_with_fee { return (0u64, 0u64); } let change = self.sum_inputs - sum_output_with_fee; if change < self.dust() { return (0u64, change) }; self.tx.outputs.push({ TransactionOutput { value: change, script_pubkey: change_script_pubkey.clone(), } }); (change, 0u64) }

borngraced · 2025-01-23T03:24:19Z

mm2src/coins/utxo/utxo_common.rs

@@ -74,7 +74,7 @@ pub const DEFAULT_SWAP_VOUT: usize = 0;
 pub const DEFAULT_SWAP_VIN: usize = 0;
 const MIN_BTC_TRADING_VOL: &str = "0.00777";

-macro_rules! true_or {
+macro_rules! return_err_if_false {


I think true_or_err is a better name

dimxy added 4 commits January 8, 2025 23:52

fix UtxoTxBuilder::build() tx fee calculation to include change output;

cea1b2f

fix use gas_fee in build(); fix min_relay_tx_fee calc; refactor break build() into functions rename fn get_tx_fee to get_fee_per_kb; fix utxos tests for recalculated tx fee

Merge branch 'dev' into fix-utxo-tx-fee-for-change

455403c

* dev: feat(tendermint): validators RPC (#2310) chore(CI): validate Cargo lock file (#2309) test(P2P): add test for peer time sync validation (#2304)

fix qrc20 gas fee calc in trade preimage

8d2fe79

revert qtum gas_fee calc in trade preimage,

bc69a42

fix refactored UtxoTxBuilder::build(): return only txfee (w/o gas fee) with tx as it used to be

dimxy added status: in progress priority: medium Moderately important tasks that should be completed but are not urgent. bug: API labels Jan 9, 2025

dimxy added status: pending review and removed status: in progress labels Jan 13, 2025

dimxy requested review from borngraced and mariocynicys January 21, 2025 17:13

onur-ozkan reviewed Jan 22, 2025

View reviewed changes

mariocynicys reviewed Jan 22, 2025

View reviewed changes

borngraced requested changes Jan 23, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(UTXO): calc of txfee with change, min relay fee #2316

fix(UTXO): calc of txfee with change, min relay fee #2316

dimxy commented Jan 9, 2025

mariocynicys commented Jan 9, 2025

dimxy commented Jan 9, 2025

borngraced commented Jan 10, 2025

dimxy commented Jan 10, 2025

onur-ozkan Jan 22, 2025

onur-ozkan Jan 22, 2025

onur-ozkan Jan 22, 2025

onur-ozkan Jan 22, 2025

onur-ozkan Jan 22, 2025

mariocynicys left a comment

mariocynicys Jan 22, 2025

mariocynicys Jan 22, 2025

mariocynicys Jan 22, 2025

mariocynicys Jan 22, 2025

mariocynicys Jan 22, 2025

mariocynicys Jan 22, 2025

mariocynicys Jan 22, 2025

mariocynicys Jan 22, 2025

borngraced left a comment

borngraced Jan 22, 2025

borngraced Jan 23, 2025 •

edited

Loading

borngraced Jan 23, 2025

borngraced Jan 23, 2025 •

edited

Loading

borngraced Jan 23, 2025

	data.fee_amount + actual_tx_fee.get_tx_fee_for_change(None)
	data.fee_amount + actual_tx_fee.get_tx_fee_for_change(0)

	data.fee_amount + fee_per_kb.get_tx_fee_for_change(Some(tx_bytes.len() as u64))
	data.fee_amount + fee_per_kb.get_tx_fee_for_change(tx_bytes.len() as u64)

		@@ -279,11 +279,43 @@ pub enum TxFee {
		pub enum ActualTxFee {

fix(UTXO): calc of txfee with change, min relay fee #2316

Are you sure you want to change the base?

fix(UTXO): calc of txfee with change, min relay fee #2316

Conversation

dimxy commented Jan 9, 2025

mariocynicys commented Jan 9, 2025

dimxy commented Jan 9, 2025

borngraced commented Jan 10, 2025

dimxy commented Jan 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mariocynicys left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

borngraced left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

borngraced Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

borngraced Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

borngraced Jan 23, 2025 •

edited

Loading

borngraced Jan 23, 2025 •

edited

Loading