Use lmdb database in picocoin #92

aido · 2016-10-08T20:32:09Z

Hi,

This pull request is the first step in introducing a simple and lightweight key-value database to picocoin:
https://symas.com/products/lightning-memory-mapped-database/

LMDB is written in C, extremely high performance and memory-efficient. It is used in other crypto currency projects such as Monero so seems like a good fit for picocoin.
Along with some other handy features LMDB can have multiple sub-databases. I haven't thought things out fully yet but blkdb could possibly be stored as a sub-database with the blockchain as another sub-database or collection of sub-databases, performance permitting.

This would add another building block to make it easier to use libccoin to build a full node if someone so wishes.

coveralls · 2016-10-08T20:55:44Z

Coverage decreased (-31.6%) to 47.814% when pulling a310547 on aido:liblmdb into 81bb7ab on jgarzik:master.

coveralls · 2016-10-08T20:55:44Z

Coverage decreased (-31.6%) to 47.814% when pulling a310547 on aido:liblmdb into 81bb7ab on jgarzik:master.

coveralls · 2016-10-08T23:32:57Z

Coverage decreased (-39.7%) to 39.763% when pulling ca60a7d on aido:liblmdb into 81bb7ab on jgarzik:master.

coveralls · 2016-10-09T00:06:14Z

Coverage decreased (-1.9%) to 77.565% when pulling f0f32f7 on aido:liblmdb into 81bb7ab on jgarzik:master.

coveralls · 2016-10-09T00:06:14Z

Coverage decreased (-1.9%) to 77.565% when pulling f0f32f7 on aido:liblmdb into 81bb7ab on jgarzik:master.

aido · 2016-12-23T13:34:56Z

Hi,

Given the preference expressed in PR #96 to using normal autoconf toolage for linking to external libraries rather than importing a tree / adding a git subtree into the codebase I assume this PR in its current form is not a good idea.

jgarzik · 2016-12-23T15:27:18Z

@aido It's a balance. As a first step we should link externally. If the library proves consensus critical, or versioning through OS proves problematic, we can look at moving in-tree via git subtree or code import. libevent is stable across most modern OS's. lmdb might(?) be different.

First step would be to link against external lmdb and change the picocoin code to use lmdb.

aido · 2016-12-23T15:32:20Z

OK Jeff,

I'll leave this PR open and play around with it a bit to see if LMDB is a good choice.

LMDB allows for the creation of multiple sub-databases so I am thinking of setting up the following sub-databases within a main mainnet/testnet database.

Database	Key	Value
blockdb	block hash	block header + list(tx_hashes)
blockheightdb	block height	block hash
txdb	tx hash	tx

Experimentation going well here: https://github.com/aido/picocoin/tree/liblmdb_test

coveralls · 2016-12-28T23:23:51Z

Coverage remained the same at 77.643% when pulling 2926183 on aido:liblmdb into 3c3d504 on jgarzik:master.

coveralls · 2016-12-28T23:23:51Z

Coverage remained the same at 77.643% when pulling 2926183 on aido:liblmdb into 3c3d504 on jgarzik:master.

coveralls · 2016-12-28T23:23:51Z

Coverage remained the same at 77.643% when pulling 2926183 on aido:liblmdb into 3c3d504 on jgarzik:master.

coveralls · 2017-01-08T02:55:35Z

Coverage remained the same at 77.643% when pulling e10840d on aido:liblmdb into 3c3d504 on jgarzik:master.

coveralls · 2017-01-08T02:55:35Z

Coverage remained the same at 77.643% when pulling e10840d on aido:liblmdb into 3c3d504 on jgarzik:master.

coveralls · 2017-01-08T02:55:35Z

Coverage remained the same at 77.643% when pulling e10840d on aido:liblmdb into 3c3d504 on jgarzik:master.

jgarzik · 2017-01-08T15:09:59Z

Looking good so far!

aido · 2017-01-08T18:34:06Z

Yes, it;s coming along nicely. LMDB seems to be quite suitable as a DB choice.

I see GitHub has a handy code review feature for each commit to a pull request so feel free to add constructive criticism if it is needed.

aido · 2017-01-09T21:41:14Z

Hi @jgarzik,

The latest commit adds a blockheightdb to speed up rebuilding of blkdb on startup.

Some of the OSX builds are failing on Travis. Travis builds fail if no output is received in 10 minutes. The blockfile test requires ~228,000 block headers to be loaded into the blockheightdb (193,000 mainnet + ~35,000 testnet). This is taking slightly more than 10 minutes on OSX due mostly to the fact that the Travis OSX VMs are painfully slow compared to the Linux VMs. There is a Travis workaround for this but it is not ideal and will probably cause the build to fail for other reasons anyway.
The mainnet test headers are in the test/hdr193000.ser file. A smaller file containing maybe 50,000 headers should do the trick.

Also, a couple of things I've noticed while testing, not really related to this PR.
brd seems to initially download 500 blocks, the getblocks limit. It then stops and just listens to the network, If restarted it will then download the next 500 blocks, reading the initial 500 from the blockdb. Is this behaviour by design? If so, without a restart brd will only ever relay blocks 0 to 500.
We now have a blockdb and a blkdb, slightly confusing/clashing names. I propose renaming blkdb to chaindb which is probably a more accurate description of what it is. Your thoughts?

coveralls · 2017-01-11T00:30:37Z

Coverage remained the same at 77.746% when pulling b773043 on aido:liblmdb into 3c3d504 on jgarzik:master.

coveralls · 2017-01-11T02:02:37Z