Consider support for osm2pgsql --append
#167
Replies: 4 comments
-
|
The major known block to using Code in question that causes the issue currently can be seen here. There will still be challenges related to post-processing SQL to work through, but from the surface it seems like those are the more minor element. |
Beta Was this translation helpful? Give feedback.
-
|
Release 0.4.4 made concrete steps to supporting Most recent documentation at this time is in docs/APPEND_MODE.md |
Beta Was this translation helpful? Give feedback.
-
|
Experimental support is in place as of version 0.4.6. See: #211 (comment) |
Beta Was this translation helpful? Give feedback.
-
Details
Currently PgOSM Flex does not support
--append. It's probably a good goal to change that. I have no real experience with using--appendso any input/help/feedback here is greatly appreciated!History
My reason for not supporting
--appendgoes back to the legacy PgOSM (pre-Flex) project. In those days, the data cleanup was 100% post-processing via a mess of Python generating SQL to transform and clean the data. The correct decision in that project was to simply import new every time, the cost of making append work was prohibitive. It's probably time to reconsider, I know plenty of users throughout the community depend on not running a full import every time.Current state
Meta table ✔️ Resolved
Resolved in #183 / #188, will be included in 0.4.0.
The most obvious issue was the
osm.pgosm_flexmeta table. The output emitted during processing before these fixes were:Post-processing can't re-run in current state
The
sql/*.sqlscripts create a mix of indexes, comments, views, functions... can't just re-run this part. But..What should happen?
My thought is that an
--appendwould add a new row inosm.pgosm_flexwith the new meta about the subsequent import.pgosm_datewill obviously (?) be different. I would guess that osm2pgsql version and pgosm-flex version should be checked and enforced to stay the same? Maybe there can be some drift, but at some point the schema will change... what to do then?This leads to the question of what needs to be done RE PgOSM Flex upgrades. When changes are made to the Lua/SQL that affect structure in the target DB, do we need to provide manual DDL for those changes too? Doing so will make those changes more expensive to make, test and maintain over time. This seems like it will be needed.
RE post-processing SQL - Those can be split out to "run once" and "always run" but that will make it hard to deploy changes to views/functions/indexes. The most realistic option I can think of immediately is to have a "pre" SQL script that clears out indexes/views/functions before importing, then re-running post-processing. That seems like a lot of additional overhead...
Beta Was this translation helpful? Give feedback.
All reactions