warn the user when using sub-optimal PBFs #601
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tilemaker's documentation suggests that users can get PBFs from BBBike:
tilemaker/docs/VECTOR_TILES.md
Line 25 in 1da4be9
BBBike uses osmconvert to generate PBFs. Reading through the osmconvert code 1, osmconvert will pack up to 31 MB of data into each block.
PBFs from Geofabrik use Osmium, which defaults to packing 8,000 objects into each block, resulting in blocks that are more like ~60 KB in size.
More, smaller blocks are better for Tilemaker:
I emailed BBBike's maintainer. He wants to one day move to Osmium, and is understandably not keen on patching osmconvert in the interim.
He did point out that a user can just run
osmium cat
on a PBF from BBBike to rejig its innards.This PR detects when a PBF is suboptimal, warns the user, and provides an explanation of how to fix it.
For one of my BBBike PBFs, this results in processing time dropping from 90 seconds to 28 seconds. (
osmium cat
itself only takes 17 seconds, and in any case, only has to be run a single time.)