Merge branch 'master' of https://github.com/interline-io/osm-planet-update

Author: irees

.circleci/config.yml

@@ -0,0 +1,16 @@
+version: 2
+jobs:
+  build:
+    docker:
+      - image: circleci/python:2.7.14-stretch
+    steps:
+      - checkout
+      - run:
+          name: Installing OSM dependencies
+          command: sudo apt-get install osmosis osmctools
+      - run:
+          name: Installing Python package and dependencies
+          command: pip install --user .
+      - run:
+          name: Running tests
+          command: python setup.py test

.editorconfig

@@ -0,0 +1,20 @@
+# http://editorconfig.org
+
+# based on https://github.com/django/django/blob/master/.editorconfig
+
+root = true
+
+[*]
+indent_style = space
+insert_final_newline = true
+trim_trailing_whitespace = true
+end_of_line = lf
+charset = utf-8
+
+# Docstrings and comments use max_line_length = 79
+[*.py]
+max_line_length = 119
+indent_size = 4
+
+[*.rb]
+indent_size = 2

.gitignore

@@ -60,3 +60,6 @@ target/
 # Virtual Environment
 venv/
 data
+
+# PyTest
+.pytest_cache/

README.md

@@ -1,15 +1,205 @@
-[![current release version](https://img.shields.io/github/release/interline-io/osm-planet-update.svg)](https://github.com/interline-io/osm-planet-update/releases)
-[![Docker Hub build status](https://img.shields.io/docker/build/interline/osm-planet-update.svg)](https://hub.docker.com/r/interline/osm-planet-update/)
+[![current release version](https://img.shields.io/github/release/interline-io/planetutils.svg)](https://github.com/interline-io/planetutils/releases)
+[![Docker Hub container image build status](https://img.shields.io/docker/build/interline/planetutils.svg)](https://hub.docker.com/r/interline/planetutils/)
+[![CircleCI code test status](https://circleci.com/gh/interline-io/planetutils.svg?style=svg)](https://circleci.com/gh/interline-io/planetutils)
 
-# OSM Planet Update Tools
+# Interline PlanetUtils
 
-Scripts and a Docker container to maintain your own mirror of the [OpenStreetMap](http://www.openstreetmap.org) planet.
+<!-- the following is generated by:
+     1. npm install --save markdown-toc -g
+     2. markdown-toc -i README.md
+-->
+
+<!-- toc -->
+
+- [Features](#features)
+- [Installation](#installation)
+  * [Using Docker container](#using-docker-container)
+  * [Using Homebrew on Mac OS](#using-homebrew-on-mac-os)
+  * [Using Python package](#using-python-package)
+- [Usage](#usage)
+  * [osm_planet_update](#osm_planet_update)
+  * [osm_planet_extract](#osm_planet_extract)
+  * [elevation_tile_download](#elevation_tile_download)
+  * [Bounding box CSV file format](#bounding-box-csv-file-format)
+- [Support](#support)
+
+<!-- tocstop -->
+
+## Features
+
+Python-based scripts and a Docker container to work with planet-scale geographic data. Using PlanetUtils, you can:
+
+- maintain your own copy of the [OpenStreetMap](http://www.openstreetmap.org) planet (by applying incremental updates)
+- cut your copy of the OSM planet into named bounding boxes (a.k.a., mini Mapzen Metro Extracts)
+- download [Mapzen Terrain Tiles from AWS](https://aws.amazon.com/public-datasets/terrain/) for the planet or your bounding boxes
+
+PlanetUtils is packaged for use as a:
+
+- Docker container, for use on any operating system
+- Python package, for use on any operating system
+- Homebrew formula, for use on Mac OS
+
+## Installation
+
+### Using Docker container
+
+Make sure you have [Docker](https://www.docker.com/community-edition) installed. Then:
+
+```sh
+docker pull interline/planetutils:release-v0.2.0
+```
+
+### Using Homebrew on Mac OS
+
+Make sure you have [Homebrew](https://brew.sh/) installed. Then:
+
+```sh
+brew install interline-io/planetutils/planetutils
+```
+
+### Using Python package
+
+If you want to install and use the Python package directly, you'll need to provide:
+
+- Python 2.x
+- Java
+- [Osmosis](https://wiki.openstreetmap.org/wiki/Osmosis)
+- [OSM C tools](https://gitlab.com/osm-c-tools/osmctools/)
+
+Then clone this repo, run the tests, and install the Python package:
+
+```sh
+git clone https://github.com/interline-io/planetutils.git
+nosetests
+pip install .
+```
 
 ## Usage
 
-Using [Osmosis](https://wiki.openstreetmap.org/wiki/Osmosis):
+PlanetUtils supplies the following command-line utilities:
+
+### osm_planet_update
+
+Update a local OSM planet. For example:
+
+- Using Docker:
+    ```sh
+    docker run --rm -v ${PWD}/data:/data -t interline/planetutils:release-v0.2.0 osm_planet_update planet-latest.osm.pbf planet-new.osm.pbf
+    ````
+- Using Homebrew or Python:
+    ```sh
+    osm_planet_update planet-latest.osm.pbf planet-new.osm.pbf
+    ```
+
+If `planet-latest.osm.pbf` does not exist, it will be downloaded, before applying updates.
+
+By default, files are downloaded from planet.openstreetmap.org. Amazon Web Services also provides OSM planets through its Public Datasets program. To instead download the planet from AWS:
+
+1. Make sure you have your [AWS credentials configured locally](http://boto3.readthedocs.io/en/latest/guide/configuration.html).
+2. Append the `--s3` flag.
+
+Note that an entire OSM planet may be upwards of 40Gb in size!
+
+For complete help on command-line arguments:
 
 ```sh
-mkdir -p data
-docker run --rm -v data:/data -t interline/osm-planet-update:release-v0.1.1 
+osm_planet_update -h
 ```
+
+### osm_planet_extract
+
+Cut up an OSM planet file into one or more extracts, defined by bounding boxes. Each bounding box is assigned a name. (This is like a mini version of Mapzen Metro Extracts!)
+
+To create a single extract:
+
+- Using Docker:
+    ```sh
+    docker run --rm -v ${PWD}/data:/data -t interline/planetutils:release-v0.2.0 osm_planet_extract /data/osm_extracts --bbox=-122.737,37.449,-122.011,37.955 --name=san-francisco
+    ````
+- Using Homebrew or Python:
+    ```sh
+    osm_planet_extract data/osm_extracts --bbox=-122.737,37.449,-122.011,37.955 --name=san-francisco
+    ```
+
+To specify more than one bounding box of tiles to download, list the bounding boxes in a [CSV file](#bounding-box). For example:
+
+- Using Docker:
+    ```sh
+    docker run --rm -v ${PWD}/data:/data -t interline/planetutils:release-v0.2.0 osm_planet_extract /data/osm_extracts --csv=/data/bboxes.csv
+    ````
+- Using Homebrew or Python:
+    ```sh
+    osm_planet_extract data/osm_extracts --csv=data/bboxes.csv
+    ```
+
+For complete help on command-line arguments:
+
+```sh
+osm_planet_extract -h
+```
+
+### elevation_tile_download
+
+Download elevation tiles from the [Terrain Tiles in the AWS Public Datasets program](https://aws.amazon.com/public-datasets/terrain/). Download for the entire planet, only tiles within a single bounding box, or within multiple bounding boxes.
+
+To download the entire planet of tiles (__which will require about 1.6Tb of space!__):
+
+- Using Docker:
+    ```sh
+    docker run --rm -v ${PWD}/data:/data -t interline/planetutils:release-v0.2.0 elevation_tile_download /data/elevation
+    ````
+- Using Homebrew or Python:
+    ```sh
+    elevation_tile_download data/elevation
+    ```
+
+To download tiles to cover a single bounding box:
+
+- Using Docker:
+    ```sh
+    docker run --rm -v ${PWD}/data:/data -t interline/planetutils:release-v0.2.0 elevation_tile_download /data/elevation --bbox=-122.737,37.449,-122.011,37.955
+    ````
+- Using Homebrew or Python:
+    ```sh
+    elevation_tile_download data/elevation --bbox=-122.737,37.449,-122.011,37.955
+    ```
+
+To specify more than one bounding box of tiles to download, list the bounding boxes in a [CSV file](#bounding-box). For example:
+
+- Using Docker:
+    ```sh
+    docker run --rm -v ${PWD}/data:/data -t interline/planetutils:release-v0.2.0 elevation_tile_download /data/elevation --csv=/data/bboxes.csv
+    ````
+- Using Homebrew or Python:
+    ```sh
+    elevation_tile_download data/elevation --csv=data/bboxes.csv
+    ```
+
+For complete help on command-line arguments:
+
+```sh
+elevation_tile_download -h
+```
+
+### Bounding box CSV file format
+<a name="bounding-box"></a>
+
+When extracting multiple bounding boxes from an OSM planet, or when downloading multiple bounding boxes of elevation tiles, you can specify your bounding boxes in a single CSV file. Do not include a header row. The format is as follows:
+
+```csv
+[name for extract],[left longitude],[bottom latitude],[right longitude],[top latitude]
+```
+
+For example:
+```csv
+san-francisco,-122.737,37.449,-122.011,37.955
+dar-es-salaam,38.894,-7.120,39.661,-6.502
+```
+
+To determine a bounding box, try the tool at http://bboxfinder.com/
+
+## Support
+
+To report a bug, please [open an issue](https://github.com/interline-io/planetutils).
+
+Interline Technologies also provides professional support and consulting services around this and other related tools. Contact us at info@interline.io for more information.

planetutils/elevation_download.py renamed to planetutils/elevation_tile_download.py

@@ -6,9 +6,9 @@
 
 def main():
     parser = argparse.ArgumentParser()
-    parser.add_argument('outpath', help='output path')
-    parser.add_argument('--csv', help='csv path with bbox definitions')
-    parser.add_argument('--bbox', help='bbox as (left,bottom,right,top)')
+    parser.add_argument('outpath', help='Output path for elevation tiles.')
+    parser.add_argument('--csv', help='Path to CSV file with bounding box definitions.')
+    parser.add_argument('--bbox', help='Bounding box for extract file. Format for coordinates: left,bottom,right,top')
     args = parser.parse_args()
     p = ElevationDownloader(args.outpath)
     if args.csv:

planetutils/planet_extract.py renamed to planetutils/osm_planet_extract.py

@@ -5,10 +5,10 @@
 
 def main():
     parser = argparse.ArgumentParser()
-    parser.add_argument('osmpath', help='osm path')
-    parser.add_argument('--csv', help='csv path with bbox definitions')
-    parser.add_argument('--name', help='extract name')
-    parser.add_argument('--bbox', help='bbox as (left,bottom,right,top)')
+    parser.add_argument('osmpath', help='Name or path to OSM planet file. Use planet_update if you do not have a copy locally.')
+    parser.add_argument('--csv', help='Path to CSV file with bounding box definitions.')
+    parser.add_argument('--name', help='Name to give to extract file.')
+    parser.add_argument('--bbox', help='Bounding box for extract file. Format for coordinates: left,bottom,right,top')
     args = parser.parse_args()
     p = Planet(args.osmpath)
     if args.csv:

planetutils/planet_update.py renamed to planetutils/osm_planet_update.py

@@ -4,9 +4,9 @@
 
 def main():
     parser = argparse.ArgumentParser()
-    parser.add_argument('osmpath', help='osm path')
-    parser.add_argument('outpath', help='output path')
-    parser.add_argument('--s3', action='store_true', help='download using s3 from osm-pds')
+    parser.add_argument('osmpath', help='Name or path to existing OSM planet file. Will be created and downloaded, if it does not exist.')
+    parser.add_argument('outpath', help='Name or path to where updated output file should be placed.')
+    parser.add_argument('--s3', action='store_true', help='Download using S3 client from AWS Public Datasets program. AWS credentials required.')
     args = parser.parse_args()
     if not os.path.exists(args.osmpath):
         print "planet does not exist; downloading"

setup.py

@@ -9,20 +9,22 @@
     long_description = f.read()
 
 setup(name='interline-planetutils',
-    version='0.1',
-    description='Interline Planet Utilities',
+    version='0.2.0',
+    description='Interline PlanetUtils',
     long_description=long_description,
-    url='https://github.com/interline-io/osm-planet-update',
+    url='https://github.com/interline-io/planetutils',
     author='Ian Rees',
     author_email='ian@interline.io',
     license='MIT',
     packages=find_packages(exclude=['contrib', 'docs', 'tests']),
-    install_requires=['boto3', 'nose'],
+    install_requires=['boto3'],
+    tests_require=['nose'],
+    test_suite = 'nose.collector',
     entry_points={
         'console_scripts': [
-            'planet_update=planetutils.planet_update:main',
-            'planet_extract=planetutils.planet_extract:main',
-            'elevation_download=planetutils.elevation_download:main'
+            'osm_planet_update=planetutils.osm_planet_update:main',
+            'osm_planet_extract=planetutils.osm_planet_extract:main',
+            'elevation_tile_download=planetutils.elevation_download:main'
         ],
     },
     classifiers=[