PRs against refactor_geometries branch or master?

[cldellow]

Hello!

I'm working on a project where I hope to continually rebuild the latest planet on a cheap Hetzner box, preferably the €30/mo 32GB CAX41 instance. The whole planet will never fit, but I'm hopeful that some number of partitions could, and then I'd stitch them together.

Towards this end, I'm exploring PRs to reduce Tilemaker memory usage. #568 was one example, and conveniently, a small patch in a part of the code that doesn't see active development.

Another straightforward one appears to be using std::vector instead of std::deque in places where inserting at both ends isn't required. Per Daniel Lemire, deque has 2x the overhead of vector. Not a huge thing, but on a 32GB box, every byte counts.

The candidates for this are in osm_store -- I'm wondering if I should make a PR against master, or against the refactor_geometries branch?

[cldellow]

Actually, this particular example is a poor one, as it looks like the affected code is the same in both branches.

But the general question still stands -- I have some thoughts on other things that I'm pretty sure have actually been changed in refactor_geometries.

[cldellow]

Heh, it turns out this particular example is an atrocious one, because according to valgrind's massif tool, there is no overhead for deque vs vector when the container type is 128 bits, as is the case for the NodeStore.

I'm a little suspicious of that, but will ignore it for now - I hope to reduce the container type size to 96 bits, at which point perhaps this will become worth revisiting.

[systemed]

Good question! The refactor_geometries branch is what'll eventually become tilemaker 3.0.

At present --store can write OSM objects (nodes/ways/relations) to disk, but not the generated geometries. This is the biggest contribution to tilemaker's RAM usage.

refactor_geometries is working towards making geometries serialisable to disk. The idea is that we can then (optionally) write geometries to an on-disk bucket, probably organised by z6 tile. We can then run the tile writing one z6 tile at a time, which means we only ever need to keep a z6 tile worth of geometries in memory.

So I'd suggest any PRs would be best aimed at refactor_geometries, but if they're something that we can use in master for now that's cool too - I'm aiming to keep the branch up-to-date with master.