-
Notifications
You must be signed in to change notification settings - Fork 231
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Planet generation issues on hardware-constrained machine #703
Comments
Don't worry about the "wrong orientation" messages, they're fairly standard and don't mean much other than for debugging geometry processing - that's why they're hidden behind
|
Tried again overnight, with little luck - unfortunately the terminal seems to have completely crashed out so I can't inspect any outputs . Inspecting the system log and cross comparing against the time that the VM disk IO dropped off a cliff, systemd-oomd seems to kill the process due to Systemd-oomd seems to be enabled by default in Ubuntu 22.04 - so it's likely that other users running on this particular OS will face similar problems during planet generation. Going to try to disable systemd-oomd and up the disk space - the process seems to have resulted in 359 mmap files (each 1GB) that aren't cleaned up, which coupled with the size of the osm.pbf (~80GB), output mbtiles (100+GB) and ubuntu itself is rapidly hitting the limit of the disk size. I'll report back on progress. |
At the risk of stating the bleeding obvious, is it perhaps worth trying a smaller extract (or several smaller extracts sequentially) and measuring resource usage when processing those, to try and see where the issues might be? |
A few hours later - it works! I bumped the storage up to 650GB, and disabled
I can't say how wise it is to leave this disabled permanently, as opposed to tweaking the internal settings - but if you're spinning up a box just for the sake of generating a planet, it's definitely better than having your hard work and processing time killed off silently and without warning. It may be worth having a troubleshooting FAQ or something - as this is a fairly insidious little issue that will catch people out for smaller machines and Ubuntu 22.04. |
Nice debugging!
Perhaps a note in a subsection of https://github.com/systemed/tilemaker/blob/master/docs/RUNNING.md#performance-and-memory-tuning would be the right spot to mention OOM killing debugging techniques for systemd-oomd? |
Attempting to generate a planet on a Ubuntu VM with 16 Cores/32GB RAM/500GB hard disk as outlined in #618 (comment), using the following command:
tilemaker --input input/planet-latest.osm.pbf --output output/osm-world.mbtiles --store /osm/tilemaker/temp/ --shard-stores --fast
Inspecting the output mbtiles results in the inbuilt tilemaker-server results in a completely blank map:
Running the same in --verbose seems to result, upon generation of the actual mbtiles file, seems to become trapped in a loop of
input multipolygon valid output multipolygon has the wrong orientation
with no further debug information. The output here is very mangled, as seen below:
Unfortunately I can't provide any more useful debug info at this time - but I'm wondering if something odd is happening in the interaction between
--fast
,--store
and--shard-stores
, as the temp folder rapidly consumes all the available disk space with mmap files.I will attempt the same again without --fast to see if the same issue arises.
The text was updated successfully, but these errors were encountered: