Merge branch 'master' of https://github.com/civicmapper/onecall-mapper

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2018 CivicMapper
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.MD

@@ -1,18 +1,87 @@
-# PA OneCall Mapper
+# PA OneCall Mapper *alpha/demo*
 
-This routine gets PA One Call tickets from an e-mail inbox publishes them to a geodata web service so they it can be viewed on a map.
+[PA OneCall](http://www.pa1call.org/PA811/Public/) has an e-mail-based notification system that will alert you of "Call Before you Dig" requests in your area (*note: CivicMapper is not affiliated with or contracted by PA OneCall*). 
 
-It's designed to listen for new One Call e-mails on one or more inboxes and publish to one or more geodata service endpoints. It checks what has been published already, and so won't re-publish/duplicate old tickets if the inbox gets messy.
+As an approved subscriber (i.e., you have to request approval from PA OneCall to get these notices), you can receive the occasional e-mail from them that looks like this:
 
-Currently:
+```
+CDC 00000 POCS MM/DD/YY TT:TT:TT 20180000000-000 NEW XCAV RTN
 
-* only Esri AGOL Feature Services are supported as endpoints;
-* it only parses and publishes the request Serial Number, Mapped Type, and Mapped Lat/Lon information from the ticket. Other information is linked in via a related table in the published service.
+============PENNSYLVANIA UNDERGROUND UTILITY LINE PROTECTION REQUEST============
+Serial Number--[20180651231]-[000] Channel#--[1117AWEB][0222]
+Message Type--[NEW][EXCAVATION][ROUTINE]
 
-We may extend the scope of the project in the future; e.g., support for other types of geodata services endpoints and formats (e.g., geojson in a GitHub repository, a table in CARTO, etc.)
+County--[ALLEGHENY] Municipality--[KILBUCK TWP]
+Work Site--[CAMP HORNE RD]
+Nearest Intersection--[OLD CAMP HORNE RD]
+Second Intersection--[HARMONY RD]
+At Intersection--[N] Between Intersections--[Y] Site Marked in White--[Y]
+Subdivision--[]
+Location Information--
+[WE ARE DOING A 10 X 5 STREET REPAIR AT 123 MAIN STREET RD FOR XYZ
+WATER AUTHORITY. LOOK FOR RECENTLY EXCAVATED &/OR TEMPORARILY PATCHED
+AREA. THIS IS WHERE WE WILL BE WORKING. CONTACT JOE PERSON @
+412-555-5555 WITH ANY QUESTIONS.]
+Caller Lat/Lon--[]
+Mapped Type--[P] Mapped Lat/Lon--
+[41.161067/-79.038070,41.538249/-80.850385,40.473058/-80.089223,
+40.517399/-80.864821,40.517905/-80.760586,40.518771/-80.0834593]
+Attachments--[http://www.pa811.org/attachments/20180651231]
+Type of Work--[STREET REPAIR] Depth--[6 FEET]
+Extent of Excavation--[VARIOUS] Method of Excavation--[DIGGING]
+Equip Type--[EXCAVATOR]
+Street--[X] Sidewalk--[ ] Pub Prop--[X] Pvt Prop--[ ] Other--[R/W-UTILITY]
+Private Front--[ ] Rear--[ ] Left--[ ] Right--[ ]
+
+Lawful Start Dates--[09-Mar-16] thru [20-Mar-16] Response Due Date--[08-Mar-16]
+Scheduled Excavation Date--[09-Mar-16] Dig Time--[0700] Duration--[ONE DAY]
+
+Caller--[JOANNE PERSON]
+Caller Phone--[412-555-5555] Caller Ext--[]
+Excavator--[BOB'S CONTRACTING]
+Address--[123 HIGHWAY RD]
+City--[PITTSBURGH] State--[PA] Zip--[12345]
+FAX--[555-555-555] Caller Type--[B]
+Email--[EMAIL@DEMO.COM]
+Work For--[A LOCAL WATER AUTHORITY]
+Person to Contact--[JOE PERSON]
+Contact Phone--[412-555-5555] Contact Ext--[]
+Best Time to Call--[0800-1700]
+Job Number--[12345/123-01]
+
+Prepared--[06-Mar-16] at [1121] by [SOMEBODAY]
+Remarks--
+[]
+
+ACP0 ACP=ALLEGH CTY PWD AL 0 AL =PPLS GAS BD 0 BD =VERIZON PA 
+DC 0 DC =DUQ LIGHT PGH IA 0 IA =BUCKEYE PT ALTN KIL0 KIL=KILBUCK TWP 
+LT 0 LT =COL GAS ROCHEST MZ 0 MZ =MCCANDLESS TSA OHO0 OHO=OTSA 
+PNZ0 PNZ=SHELL APPALACH WX 0 WX =WVWA 
+
+Serial Number--[20180651231]-[000]
+========== Copyright (c) 2018 by Pennsylvania One Call System, Inc. ==========
+```
+
+This set of scripts will can take a PA One Call e-mails and publish them to a standard geodata web service endpoint so the tickets can be viewed on a web map. Currently:
+
+* it only parses and publishes the request Serial Number, Mapped Type, and Mapped Lat/Lon information from the ticket; other information could easily be parsed and stored as well
+* it checks what has been published already, and so won't re-publish/duplicate old tickets (if the inbox gets messy)
+* only Esri AGOL Feature Services are supported as endpoints right now; however it would be trivial to store and publish the data in other formats, (e.g., a `csv` or `geojson` in a GitHub repository, a table in CARTO, etc.)
+
+For now, this is just an experiment and technology demo; CivicMapper is not affiliated with or contracted by PA OneCall.
 
 ## Software
 
 This is built with Python 3.6 and uses the Python Requests library. We use `pipenv` for a python package and virtual environment management.
 
-We run it as a CRON job every six hours; it could also easily be run as a Windows task.
+The [`config.py`](https://github.com/civicmapper/onecall-mapper/blob/master/config.example.py) file is used to specify e-mails, data endpoints, and authentication information so that the script can get what it needs to get and publish what it's intended to publish.
+
+When run, the script:
+
+1. logs into your IMAP-based e-mail inbox and searches for messages from the configured address (default: `Delivery@pa1call.net`)
+2. loads any e-mails matching the sender and subject from PA One call into memory
+3. uses a series of regex statements on each e-mail to parse out desired info
+4. checks the existing publication endpoint, to see if any of the notices from the inbox have already been published
+5. publishes new notices to the publication endpoint.
+
+It's designed to be run as a scheduled task periodically (e.g., a `cron` job or a Windows scheduled task).