Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Branch and Bound algorithm #13

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 143 additions & 0 deletions branchandbound.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
var utils = require('./utils')

var maxTries = 1000000
Copy link

@murchandamus murchandamus Jul 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My gut feeling was 100k, you have 1M here. That might be very long on a huge wallet in an unfortunate UTXO pool and target combination. Might want to check what amount of time that allows, especially since JavaScript is probably slower than C++ in evaluation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A typo I think! Thanks

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but yeah 100k should be enough

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have left 1M here; in my experience it's very fast anyway, even in javascript on browser/in node


function calculateEffectiveValues (utxos, feeRate) {
return utxos.map(function (utxo) {
if (isNaN(utils.uintOrNaN(utxo.value))) {
return {
utxo: utxo,
effectiveValue: 0
}
}

var effectiveFee = utils.inputBytes(utxo) * feeRate
var effectiveValue = utxo.value - effectiveFee
return {
utxo: utxo,
effectiveValue: effectiveValue
}
})
}

module.exports = function branchAndBound (utxos, outputs, feeRate, factor) {
if (!isFinite(utils.uintOrNaN(feeRate))) return {}

var costPerOutput = utils.outputBytes({}) * feeRate
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: as written, this does not take segwit/different input/ouptut sizes into account

var costPerInput = utils.inputBytes({}) * feeRate
var costOfChange = Math.floor((costPerInput + costPerOutput) * factor)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't take segwit into account; but neither does utils.finalize


var outAccum = utils.sumOrNaN(outputs) + utils.transactionBytes([], outputs) * feeRate

var effectiveUtxos = calculateEffectiveValues(utxos, feeRate).filter(function (x) {
return x.effectiveValue > 0
}).sort(function (a, b) {
return b.effectiveValue - a.effectiveValue
})

var selected = search(effectiveUtxos, outAccum, costOfChange)
if (selected != null) {
Copy link
Contributor

@dcousens dcousens Jul 6, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

!==, I'll add standard-js to the repo to avoid style nits

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

standard-js is already in there.

!= null is fine for it, since it checks both undefined and null. Maybe it would be better to somehow return "empty" value from the "inner" function more explicitly (to avoid "null !== undefined" shenanigans), than just return;?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I originally thought about throwing Errors, but that's also kind of ugly

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will leave this like this

var inputs = []

for (var i = 0; i < effectiveUtxos.length; i++) {
if (selected[i]) {
inputs.push(effectiveUtxos[i].utxo)
}
}

return utils.finalize(inputs, outputs, feeRate)
} else {
const fee = feeRate * utxos.reduce(function (a, x) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just in case this isn't happening elsewhere, be sure that you don't forget accounting for the fees of the transaction overhead of 10 bytes and requested outputs to the target of the selection!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm. Where is that hapenning in the achow101 PR code? I don't see it. (But maybe it happens before it goes into wallet.cpp CWallet::SelectCoinsMinConf)

Copy link
Contributor Author

@karelbilek karelbilek Jul 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, the C++ doesn's seem to do that either.

It is not done in the coinselection.cpp - https://github.com/achow101/bitcoin/blob/19761499d5657b1f272a2fb04185dacf218fb104/src/wallet/coinselection.cpp

And it's not done in wallet.cpp - here the nvalue is just sum of the outputs
https://github.com/achow101/bitcoin/blob/19761499d5657b1f272a2fb04185dacf218fb104/src/wallet/wallet.cpp#L2437

nvaluetoselect is just nvalue here (nFeeRet is 0 in the first pass, and BnB is used only in the first pass)
https://github.com/achow101/bitcoin/blob/19761499d5657b1f272a2fb04185dacf218fb104/src/wallet/wallet.cpp#L2503

Here the SelectCoins is called, which calls (through some layers) BnB with the same target
https://github.com/achow101/bitcoin/blob/19761499d5657b1f272a2fb04185dacf218fb104/src/wallet/wallet.cpp#L2555

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's actually a bug in the c++ code, I have added a comment to the PR, we'll see

Copy link
Contributor Author

@karelbilek karelbilek Jul 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bug in the c++ code, now corrected. Anyway though, I noticed something weird.

When I corrected the error and add the overhead, the algorithm performs slightly worse. When I run, on the same randomly generated data, both the BnB with incorrect (lower) target and BnB with correct target, the BnB with incorrect targets always has better results. The difference is small, but consistent.

I have no idea why.

I will anyway add some tests finally, maybe there is some other subtle bug that I haven't found.

Copy link
Contributor Author

@karelbilek karelbilek Jul 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oooh the issue is in the simulation, I think it allows negative fees. Which is what happens. no I was wrong

Copy link

@murchandamus murchandamus Jul 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be that now that the transaction size is properly estimated, the allowance window for the exact match actually is too big. You might want to try with a "costOfChange" that is actually only the saved change output's cost (or just discount the cost of the saved input). That would be the lower bound on the saved cost, but may cause the algorithm to find fewer matches or require more inputs to find matches.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can replicate the results with your code too -
murchandamus/CoinSelectionSimulator#5

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[for other readers: the discussion continued into the bitcoind pull request]

return a + utils.inputBytes(x)
}, 0)

return {
fee: fee
}
}
}

// Depth first search
// Inclusion branch first (Largest First Exploration), then exclusion branch
function search (effectiveUtxos, target, costOfChange) {
if (effectiveUtxos.length === 0) {
return
}

var tries = maxTries

var selected = [] // true -> select the utxo at this index
var selectedAccum = 0 // sum of effective values

var traversingExclusion = [] // true -> traversing exclusion branch at this index

var done = false
var backtrack = false

var remaining = effectiveUtxos.reduce(function (a, x) {
return a + x.effectiveValue
}, 0)

var depth = 0
while (!done) {
if (tries <= 0) { // Too many tries, exit
return
} else if (selectedAccum > target + costOfChange) { // Selected value is out of range, go back and try other branch
backtrack = true
} else if (selectedAccum >= target) { // Selected value is within range
done = true
} else if (depth >= effectiveUtxos.length) { // Reached a leaf node, no solution here
backtrack = true
} else if (selectedAccum + remaining < target) { // Cannot possibly reach target with amount remaining
if (depth === 0) { // At the first utxo, no possible selections, so exit
return
} else {
backtrack = true
}
} else { // Continue down this branch
// Remove this utxo from the remaining utxo amount
remaining -= effectiveUtxos[depth].effectiveValue
// Inclusion branch first (Largest First Exploration)
selected[depth] = true
selectedAccum += effectiveUtxos[depth].effectiveValue
depth++
}

// Step back to the previous utxo and try the other branch
if (backtrack) {
backtrack = false // Reset
depth--

// Walk backwards to find the first utxo which has not has its second branch traversed
while (traversingExclusion[depth]) {
// Reset this utxo's selection
if (selected[depth]) {
selectedAccum -= effectiveUtxos[depth].effectiveValue
}
selected[depth] = false
traversingExclusion[depth] = false
remaining += effectiveUtxos[depth].effectiveValue

// Step back one
depth--

if (depth < 0) { // We have walked back to the first utxo and no branch is untraversed. No solution, exit.
return
}
}

if (!done) {
// Now traverse the second branch of the utxo we have arrived at.
traversingExclusion[depth] = true

// These were always included first, try excluding now
selected[depth] = false
selectedAccum -= effectiveUtxos[depth].effectiveValue
depth++
}
}
tries--
}

return selected
}
85 changes: 85 additions & 0 deletions stats/strategies.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
let accumulative = require('../accumulative')
let branchandbound = require('../branchandbound')
let blackjack = require('../blackjack')
let shuffle = require('fisher-yates')
let shuffleInplace = require('fisher-yates/inplace')
Expand Down Expand Up @@ -40,6 +41,68 @@ function blackrand (utxos, outputs, feeRate) {
return accumulative(utxos, outputs, feeRate)
}

function bnbrand (utxos, outputs, feeRate, factor) {
// attempt to use the bnb strategy first (no change output)
let base = branchandbound(utxos, outputs, feeRate, factor)
if (base.inputs) return base

utxos = shuffle(utxos)

// else, try the accumulative strategy
return accumulative(utxos, outputs, feeRate)
}

function bnbmin (utxos, outputs, feeRate, factor) {
// attempt to use the blackjack strategy first (no change output)
let base = branchandbound(utxos, outputs, feeRate, factor)
if (base.inputs) return base

// order by descending value
utxos = utxos.concat().sort((a, b) => b.value - a.value)

// else, try the accumulative strategy
return accumulative(utxos, outputs, feeRate)
}

function bnbmax (utxos, outputs, feeRate, factor) {
// attempt to use the bnb strategy first (no change output)
let base = branchandbound(utxos, outputs, feeRate, factor)
if (base.inputs) return base

// order by ascending value
utxos = utxos.concat().sort((a, b) => a.value - b.value)

// else, try the accumulative strategy
return accumulative(utxos, outputs, feeRate)
}

function bnbcs (utxos, outputs, feeRate, factor) {
// attempt to use the bnb strategy first (no change output)
let base = branchandbound(utxos, outputs, feeRate, factor)
if (base.inputs) return base

// else, try the current default
return coinSelect(utxos, outputs, feeRate)
}

function bnbus (utxos, outputs, feeRate, factor) {
// order by descending value, minus the inputs approximate fee
function utxoScore (x, feeRate) {
return x.value - (feeRate * utils.inputBytes(x))
}

// attempt to use the blackjack strategy first (no change output)
let base = branchandbound(utxos, outputs, feeRate, factor)
if (base.inputs) return base

utxos = utxos.concat().sort(function (a, b) {
return utxoScore(b, feeRate) - utxoScore(a, feeRate)
})

// else, try the accumulative strategy
return accumulative(utxos, outputs, feeRate)
}

function maximal (utxos, outputs, feeRate) {
utxos = utxos.concat().sort((a, b) => a.value - b.value)

Expand Down Expand Up @@ -125,9 +188,19 @@ function privet (utxos, outputs, feeRate) {
return accumulative(utxos, outputs, feeRate)
}

function useBnbWithFactor (strategy, factor) {
return (utxos, outputs, feeRate) => strategy(utxos, outputs, feeRate, factor)
}

module.exports = {
accumulative,
bestof,
bnb: useBnbWithFactor(branchandbound, 0.5),
bnbrand: useBnbWithFactor(bnbrand, 0.5),
bnbmin: useBnbWithFactor(bnbmin, 0.5),
bnbmax: useBnbWithFactor(bnbmax, 0.5),
bnbcs: useBnbWithFactor(bnbcs, 0.5),
bnbus: useBnbWithFactor(bnbus, 0.5),
blackjack,
blackmax,
blackmin,
Expand All @@ -140,3 +213,15 @@ module.exports = {
proximal,
random
}

// uncomment for benchmarking bnb parameters
// let res = {}
//
// for (let i = 0; i <= 200; i+=1) {
// let factor = i / 100;
// res['M' + i] = useBnbWithFactor(bnbmin, factor)
// res['R' + i] = useBnbWithFactor(bnbrand, factor)
// res['R' + i] = (u, o, f) => bnbrand(u, o, f, factor)
// }
//
// module.exports = res