Introduce `evicted-at`/`last-evicted` timestamps #1839

evanlinjin · 2025-02-16T09:13:28Z

Partially Fixes #1740.
Replaces #1765.
Replaces #1811.

Description

This PR allows the receiving structures (bdk_chain, bdk_wallet) to detect and evict incoming transactions that are double spent (cancelled).

We add a new field to TxUpdate (TxUpdate::evicted_ats), which in turn, updates the last_evicted timestamps that are tracked/persisted by TxGraph. This is similar to how TxUpdate::seen_ats update the last_seen timestamp in TxGraph. Transactions with a last_evicted timestamp higher than their last_seen timestamp are considered evicted.

SpkWithExpectedTxids is introduced in SpkClient to track expected Txids for each spk. During a sync, if any Txids from SpkWithExpectedTxids are not in the current history of an spk obtained from the chain source, those Txids are considered missing. Support for SpkWithExpectedTxids has been added to both bdk_electrum and bdk_esplora chain source crates.

The canonicalization algorithm is updated to disregard transactions with a last_evicted timestamp greater than or equal to their last_seen timestamp, except in cases where transitivity rules apply.

Notes to the reviewers

This PR does not fix #1740 for block-by-block chain source (such as bdk_bitcoind_rpc). This work is done in a separate PR (#1857).

Changelog notice

Add TxUpdate::evicted_ats which tracks transactions that have been replaced and are no longer present in mempool.
Change TxGraph to track last_evicted timestamps. This is when a transaction is last marked as missing from the mempool.
The canonicalization algorithm now disregards transactions with a last_evicted timestamp greater than or equal to it's last_seen timestamp, except when a canonical descendant exists due to rules of transitivity.
Add SpkWithExpectedTxids in spk_client which keeps track of expected Txids for each spk.
Change bdk_electrum and bdk_esplora to understand SpkWithExpectedTxids.
Add SyncRequestBuilder::expected_txids_of_spk method which adds an association between txids and spks.

Checklists

All Submissions:

I've signed all my commits
I followed the contribution guidelines
I ran cargo fmt and cargo clippy before committing

New Features:

I've added tests for the new feature
I've added docs for the new feature

Bugfixes:

This pull request breaks the existing API
I've added tests to reproduce the issue which are now passing
I'm linking the issue being fixed by this PR

nymius

cACK 9d6c795

This unlocks functionality on demand and I think the approach is OK. Tests are thorough enough.

crates/chain/src/tx_graph.rs

crates/core/src/spk_client.rs

crates/esplora/tests/async_ext.rs

crates/esplora/tests/blocking_ext.rs

This branch includes: - bitcoindevkit/bdk#1855 - bitcoindevkit/bdk#1839 - the patch for esplora-client

LagginTimes

Nit: Minor typos and grammar fixes.

crates/core/src/spk_client.rs

crates/chain/src/tx_graph.rs

This is a set of evicted txs from the mempool.

The evicted-at and last-evicted timestamp informs the `TxGraph` when the transaction was last deemed as missing (evicted) from the mempool. The canonicalization algorithm is changed to disregard transactions with a last-evicted timestamp greater or equal to it's last-seen timestamp. The exception is when we have a canonical descendant due to rules of transitivity. Update rusqlite_impl to persist `last_evicted`. Also update docs: * Remove duplicate paragraphs about `ChangeSet`s. * Add "Canonicalization" section which expands on methods that require canonicalization and the associated data types used in the canonicalization algorithm.

The spk history returned from Electrum should have these txs present. Any missing tx will be considered evicted from the mempool.

* `TxGraph::try_list_expected_spk_txids` * `TxGraph::list_expected_spk_txids` * `IndexedTxGraph::try_list_expected_spk_txids` * `IndexedTxGraph::list_expected_spk_txids` Co-authored-by: valued mammal <valuedmammal@protonmail.com> Co-authored-by: Wei Chen <wzc110@gmail.com>

Co-authored-by: Wei Chen <wzc110@gmail.com>

Also add `detect_receive_tx_cancel` test. Also rm `miniscript` dependency. Update ci to rm `miniscript/no-std` for `bdk_esplora`. Co-authored-by: Wei Chen <wzc110@gmail.com>

notmandatory

ut ACK 1eb0c32

Thanks for the fix and all the new docs, looks like a solid solution. Before (or after) this goes in it would be nice to have some confirmation from @ErikDeSmedt that it fixes #1740 for him (at least using electrum and esplora).

notmandatory · 2025-03-07T17:55:17Z

.github/workflows/cont_integration.yml

@@ -94,7 +94,7 @@ jobs:
      - name: Check esplora
        working-directory: ./crates/esplora
        # TODO "--target thumbv6m-none-eabi" should work but currently does not
-        run: cargo check --no-default-features --features miniscript/no-std,bdk_chain/hashbrown
+        run: cargo check --no-default-features --features bdk_chain/hashbrown


👍 thanks for cleaning this up!

It raised my attention, shouldn't we be using bdk_core/hashbrown instead ?

notmandatory · 2025-03-07T18:06:59Z

crates/chain/src/tx_graph.rs

-//! methods only consider transactions that are "canonical" (i.e., in the best chain or in mempool).
-//! We decide which transactions are canonical based on the transaction's anchors and the
-//! `last_seen` (as unconfirmed) timestamp.
+//! # Canonicalization


👍 Good to see these new docs as this is core to how the the tx_graph logic works.

notmandatory · 2025-03-07T18:21:24Z

crates/chain/src/tx_graph.rs

@@ -1107,6 +1245,9 @@ pub struct ChangeSet<A = ()> {
    pub anchors: BTreeSet<(A, Txid)>,
    /// Added last-seen unix timestamps of transactions.
    pub last_seen: BTreeMap<Txid, u64>,
+    /// Added timestamps of when a transaction is last evicted from the mempool.
+    #[cfg_attr(feature = "serde", serde(default))]


Is this cfg_attr only needed to load data from a pre-v2 serialized file?

Yes, this ensures backwards compatibility of our serialized changeset (as we promised).

oleonardolima

cACK 1eb0c32

Great work! I left some minor comments/questions that if applicable could be done in a follow-up PR.

oleonardolima · 2025-03-07T19:04:47Z

crates/core/src/tx_update.rs

+    /// When transactions were discovered to be missing (evicted) from the mempool.
+    ///
+    /// [`SyncRequest::start_time`](crate::spk_client::SyncRequest::start_time) can be used to
+    /// provide the `seen_at` value.


nit:

Suggested change

/// provide the `seen_at` value.

/// provide the `evicted_at` value.

It can be used to provide both actually. Will change.

oleonardolima · 2025-03-07T20:13:10Z

crates/esplora/tests/common/mod.rs

+const PK_BYTES: &[u8] = &[
+    12, 244, 72, 4, 163, 4, 211, 81, 159, 82, 153, 123, 125, 74, 142, 40, 55, 237, 191, 231, 31,
+    114, 89, 165, 83, 141, 8, 203, 93, 240, 53, 101,
+];
+
+#[allow(dead_code)]
+pub fn get_test_spk() -> ScriptBuf {
+    let secp = Secp256k1::new();
+    let pk = UntweakedPublicKey::from_slice(PK_BYTES).expect("Must be valid PK");
+    ScriptBuf::new_p2tr(&secp, pk, None)
+}


nit: as it's common between bdk_esplora and bdk_electrum too, couldn't it be moved to bdk_testenv instead?

oleonardolima · 2025-03-07T20:22:41Z

crates/core/src/spk_client.rs

+    /// Iterate over [`ScriptBuf`]s with corresponding [`Txid`] histories contained in this request.
+    pub fn iter_spks_with_expected_txids(
+        &mut self,
+    ) -> impl ExactSizeIterator<Item = SpkWithExpectedTxids> + '_ {
+        SyncIter::<I, SpkWithExpectedTxids>::new(self)
+    }


question: the iter_spks fn above is not being used anymore, at least for SyncRequest, do we need to keep it?

oleonardolima · 2025-03-07T20:24:07Z

.github/workflows/cont_integration.yml

@@ -94,7 +94,7 @@ jobs:
      - name: Check esplora
        working-directory: ./crates/esplora
        # TODO "--target thumbv6m-none-eabi" should work but currently does not
-        run: cargo check --no-default-features --features miniscript/no-std,bdk_chain/hashbrown
+        run: cargo check --no-default-features --features bdk_chain/hashbrown


It raised my attention, shouldn't we be using bdk_core/hashbrown instead ?

ValuedMammal · 2025-03-08T01:03:58Z

crates/electrum/src/bdk_electrum_client.rs

+            let (spks, expected_spk_txids): (Vec<(u32, ScriptBuf)>, HashMap<ScriptBuf, _>) = spks
+                .iter()
+                .map(|(spk_i, spk_with_expected_txids)| {
+                    (
+                        (*spk_i, spk_with_expected_txids.spk.clone()),
+                        (
+                            spk_with_expected_txids.spk.clone(),
+                            spk_with_expected_txids.expected_txids.clone(),
+                        ),
+                    )
+                })
+                .unzip();


I'm curious about the reasoning behind this

About what exactly? What the two variables are used for?

why split the batch of spks into two new containers?

evanlinjin force-pushed the evicted_at branch 5 times, most recently from 6dc26c4 to 0fcc097 Compare February 21, 2025 10:48

LagginTimes force-pushed the evicted_at branch 2 times, most recently from 540a83f to 5107b3a Compare February 26, 2025 14:26

evanlinjin force-pushed the evicted_at branch 6 times, most recently from 5718a3e to 07903a8 Compare February 28, 2025 03:34

evanlinjin changed the title ~~Temporary PR: Evicted at~~ Introduce evicted-at/last-evicted timestamps Feb 28, 2025

evanlinjin marked this pull request as ready for review February 28, 2025 04:00

evanlinjin mentioned this pull request Feb 28, 2025

Introduce evicted-at/last-evicted timestamps #1811

Closed

8 tasks

evanlinjin assigned evanlinjin and LagginTimes Feb 28, 2025

evanlinjin added bug Something isn't working api A breaking API change labels Feb 28, 2025

evanlinjin requested review from LLFourn, nymius and LagginTimes February 28, 2025 04:01

evanlinjin force-pushed the evicted_at branch from 07903a8 to 9d6c795 Compare February 28, 2025 04:03

evanlinjin mentioned this pull request Feb 28, 2025

feat(rpc)!: Update Emitter::mempool to support evicted_at #1857

Draft

8 tasks

evanlinjin requested a review from notmandatory March 2, 2025 06:13

nymius reviewed Mar 6, 2025

View reviewed changes

stevenroose added a commit to ark-bitcoin/bark that referenced this pull request Mar 6, 2025

Set bdk dependency to a custom dev branch

c1c2103

This branch includes: - bitcoindevkit/bdk#1855 - bitcoindevkit/bdk#1839 - the patch for esplora-client

LagginTimes reviewed Mar 6, 2025

View reviewed changes

crates/core/src/spk_client.rs Outdated Show resolved Hide resolved

crates/chain/src/tx_graph.rs Outdated Show resolved Hide resolved

crates/chain/src/tx_graph.rs Outdated Show resolved Hide resolved

feat(core): Add TxUpdate::evicted_ats

c90e1f2

This is a set of evicted txs from the mempool.

evanlinjin force-pushed the evicted_at branch 3 times, most recently from c11515d to 6afb703 Compare March 7, 2025 05:19

evanlinjin and others added 3 commits March 7, 2025 16:22

feat(core): Add expected txids to SyncRequest spks

9f2fc06

The spk history returned from Electrum should have these txs present. Any missing tx will be considered evicted from the mempool.

evanlinjin force-pushed the evicted_at branch 5 times, most recently from c23120c to e538e66 Compare March 7, 2025 06:28

evanlinjin and others added 2 commits March 7, 2025 17:59

feat(electrum): Handle spks with expected txids

08116fa

Co-authored-by: Wei Chen <wzc110@gmail.com>

feat(esplora): Handle spks with expected txids

1eb0c32

Also add `detect_receive_tx_cancel` test. Also rm `miniscript` dependency. Update ci to rm `miniscript/no-std` for `bdk_esplora`. Co-authored-by: Wei Chen <wzc110@gmail.com>

evanlinjin force-pushed the evicted_at branch from e538e66 to 1eb0c32 Compare March 7, 2025 07:00

notmandatory approved these changes Mar 7, 2025

View reviewed changes

oleonardolima reviewed Mar 7, 2025

View reviewed changes

ValuedMammal reviewed Mar 8, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce `evicted-at`/`last-evicted` timestamps #1839

Introduce `evicted-at`/`last-evicted` timestamps #1839

evanlinjin commented Feb 16, 2025 •

edited

Loading

nymius left a comment •

edited

Loading

LagginTimes left a comment

notmandatory left a comment

notmandatory Mar 7, 2025

oleonardolima Mar 7, 2025

notmandatory Mar 7, 2025

notmandatory Mar 7, 2025

evanlinjin Mar 7, 2025

oleonardolima left a comment

oleonardolima Mar 7, 2025

evanlinjin Mar 7, 2025

oleonardolima Mar 7, 2025

oleonardolima Mar 7, 2025

oleonardolima Mar 7, 2025

ValuedMammal Mar 8, 2025

evanlinjin Mar 9, 2025

ValuedMammal Mar 9, 2025

	/// provide the `seen_at` value.
	/// provide the `evicted_at` value.

Introduce evicted-at/last-evicted timestamps #1839

Are you sure you want to change the base?

Introduce evicted-at/last-evicted timestamps #1839

Conversation

evanlinjin commented Feb 16, 2025 • edited Loading

Description

Notes to the reviewers

Changelog notice

Checklists

All Submissions:

New Features:

Bugfixes:

nymius left a comment • edited Loading

Choose a reason for hiding this comment

LagginTimes left a comment

Choose a reason for hiding this comment

notmandatory left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleonardolima left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Introduce `evicted-at`/`last-evicted` timestamps #1839

Introduce `evicted-at`/`last-evicted` timestamps #1839

evanlinjin commented Feb 16, 2025 •

edited

Loading

nymius left a comment •

edited

Loading