From fa401c1a704c486ad66e4e86538351687241be5a Mon Sep 17 00:00:00 2001 From: Christian Gass Date: Mon, 26 Mar 2018 14:14:50 -0400 Subject: [PATCH 1/3] Update README.MD --- README.MD | 83 ++++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 76 insertions(+), 7 deletions(-) diff --git a/README.MD b/README.MD index 70da4e8..672f967 100644 --- a/README.MD +++ b/README.MD @@ -1,18 +1,87 @@ -# PA OneCall Mapper +# PA OneCall Mapper *alpha/demo* -This routine gets PA One Call tickets from an e-mail inbox publishes them to a geodata web service so they it can be viewed on a map. +[PA OneCall](http://www.pa1call.org/PA811/Public/) has an e-mail-based notification system that will alert you of "Call Before you Dig" requests in your area (*note: CivicMapper is not affiliated with or contracted by PA OneCall*) As an approved subscriber (i.e., you have to request approval from PA OneCall), you can receive the occasional e-mail from them that looks like this: -It's designed to listen for new One Call e-mails on one or more inboxes and publish to one or more geodata service endpoints. It checks what has been published already, and so won't re-publish/duplicate old tickets if the inbox gets messy. +``` +CDC 00000 POCS MM/DD/YY TT:TT:TT 20180000000-000 NEW XCAV RTN + +============PENNSYLVANIA UNDERGROUND UTILITY LINE PROTECTION REQUEST============ +Serial Number--[20180651231]-[000] Channel#--[1117AWEB][0222] +Message Type--[NEW][EXCAVATION][ROUTINE] + +County--[ALLEGHENY] Municipality--[KILBUCK TWP] +Work Site--[CAMP HORNE RD] +Nearest Intersection--[OLD CAMP HORNE RD] +Second Intersection--[HARMONY RD] +At Intersection--[N] Between Intersections--[Y] Site Marked in White--[Y] +Subdivision--[] +Location Information-- +[WE ARE DOING A 10 X 5 STREET REPAIR AT 123 MAIN STREET RD FOR XYZ +WATER AUTHORITY. LOOK FOR RECENTLY EXCAVATED &/OR TEMPORARILY PATCHED +AREA. THIS IS WHERE WE WILL BE WORKING. CONTACT JOE PERSON @ +412-555-5555 WITH ANY QUESTIONS.] +Caller Lat/Lon--[] +Mapped Type--[P] Mapped Lat/Lon-- +[41.161067/-79.038070,41.538249/-80.850385,40.473058/-80.089223, +40.517399/-80.864821,40.517905/-80.760586,40.518771/-80.0834593] +Attachments--[http://www.pa811.org/attachments/20180651231] +Type of Work--[STREET REPAIR] Depth--[6 FEET] +Extent of Excavation--[VARIOUS] Method of Excavation--[DIGGING] +Equip Type--[EXCAVATOR] +Street--[X] Sidewalk--[ ] Pub Prop--[X] Pvt Prop--[ ] Other--[R/W-UTILITY] +Private Front--[ ] Rear--[ ] Left--[ ] Right--[ ] + +Lawful Start Dates--[09-Mar-16] thru [20-Mar-16] Response Due Date--[08-Mar-16] +Scheduled Excavation Date--[09-Mar-16] Dig Time--[0700] Duration--[ONE DAY] + +Caller--[JOANNE PERSON] +Caller Phone--[412-555-5555] Caller Ext--[] +Excavator--[BOB'S CONTRACTING] +Address--[123 HIGHWAY RD] +City--[PITTSBURGH] State--[PA] Zip--[12345] +FAX--[555-555-555] Caller Type--[B] +Email--[EMAIL@DEMO.COM] +Work For--[A LOCAL WATER AUTHORITY] +Person to Contact--[JOE PERSON] +Contact Phone--[412-555-5555] Contact Ext--[] +Best Time to Call--[0800-1700] +Job Number--[12345/123-01] + +Prepared--[06-Mar-16] at [1121] by [SOMEBODAY] +Remarks-- +[] + +ACP0 ACP=ALLEGH CTY PWD AL 0 AL =PPLS GAS BD 0 BD =VERIZON PA +DC 0 DC =DUQ LIGHT PGH IA 0 IA =BUCKEYE PT ALTN KIL0 KIL=KILBUCK TWP +LT 0 LT =COL GAS ROCHEST MZ 0 MZ =MCCANDLESS TSA OHO0 OHO=OTSA +PNZ0 PNZ=SHELL APPALACH WX 0 WX =WVWA + +Serial Number--[20180651231]-[000] +========== Copyright (c) 2018 by Pennsylvania One Call System, Inc. ========== +``` + +This set of scripts will can take a PA One Call e-mails and publish them to a standard geodata web service endpoint so the tickets can be viewed on a web map. Currently: -* only Esri AGOL Feature Services are supported as endpoints; -* it only parses and publishes the request Serial Number, Mapped Type, and Mapped Lat/Lon information from the ticket. Other information is linked in via a related table in the published service. +* it only parses and publishes the request Serial Number, Mapped Type, and Mapped Lat/Lon information from the ticket; other information could be parsed and stored with it, or linked via a related table in the published service +* it checks what has been published already, and so won't re-publish/duplicate old tickets if the inbox gets messy +* only Esri AGOL Feature Services are supported as endpoints; however it would be trivial to store and publish the data in other formats, (e.g., a csv or geojson in a GitHub repository, a table in CARTO, etc.) -We may extend the scope of the project in the future; e.g., support for other types of geodata services endpoints and formats (e.g., geojson in a GitHub repository, a table in CARTO, etc.) +For now, this is just an experiment and technology demo; CivicMapper is not affiliated with or contracted by PA OneCall. ## Software This is built with Python 3.6 and uses the Python Requests library. We use `pipenv` for a python package and virtual environment management. -We run it as a CRON job every six hours; it could also easily be run as a Windows task. +The [`config.py`](https://github.com/civicmapper/onecall-mapper/blob/master/config.example.py) file is used to specify e-mails, data endpoints, and authentication information so that the script can get what it needs to get and publish what it's intended to publish. + +When run, the script: + +1. logs into your IMAP-based e-mail inbox and searches for messages from the configured address (default: `Delivery@pa1call.net`) +2. loads any e-mails matching the sender and subject from PA One call into memory +3. uses a series of regex statements on each e-mail to parse out desired info +4. checks the existing publication endpoint, to see if any of the notices from the inbox have already been published +5. publishes new notices to the publication endpoint. + +It's designed to be run as a scheduled task periodically (e.g., a `cron` job or a Windows scheduled task). From ae969bce5bdb70e3df76ce5495b8938a81d4e14b Mon Sep 17 00:00:00 2001 From: Christian Gass Date: Mon, 26 Mar 2018 14:17:29 -0400 Subject: [PATCH 2/3] Update README.MD --- README.MD | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/README.MD b/README.MD index 672f967..14252f1 100644 --- a/README.MD +++ b/README.MD @@ -1,6 +1,8 @@ # PA OneCall Mapper *alpha/demo* -[PA OneCall](http://www.pa1call.org/PA811/Public/) has an e-mail-based notification system that will alert you of "Call Before you Dig" requests in your area (*note: CivicMapper is not affiliated with or contracted by PA OneCall*) As an approved subscriber (i.e., you have to request approval from PA OneCall), you can receive the occasional e-mail from them that looks like this: +[PA OneCall](http://www.pa1call.org/PA811/Public/) has an e-mail-based notification system that will alert you of "Call Before you Dig" requests in your area (*note: CivicMapper is not affiliated with or contracted by PA OneCall*). + +As an approved subscriber (i.e., you have to request approval from PA OneCall to get these notices), you can receive the occasional e-mail from them that looks like this: ``` CDC 00000 POCS MM/DD/YY TT:TT:TT 20180000000-000 NEW XCAV RTN @@ -60,13 +62,11 @@ Serial Number--[20180651231]-[000] ========== Copyright (c) 2018 by Pennsylvania One Call System, Inc. ========== ``` -This set of scripts will can take a PA One Call e-mails and publish them to a standard geodata web service endpoint so the tickets can be viewed on a web map. - -Currently: +This set of scripts will can take a PA One Call e-mails and publish them to a standard geodata web service endpoint so the tickets can be viewed on a web map. Currently: -* it only parses and publishes the request Serial Number, Mapped Type, and Mapped Lat/Lon information from the ticket; other information could be parsed and stored with it, or linked via a related table in the published service -* it checks what has been published already, and so won't re-publish/duplicate old tickets if the inbox gets messy -* only Esri AGOL Feature Services are supported as endpoints; however it would be trivial to store and publish the data in other formats, (e.g., a csv or geojson in a GitHub repository, a table in CARTO, etc.) +* it only parses and publishes the request Serial Number, Mapped Type, and Mapped Lat/Lon information from the ticket; other information could easily be parsed and stored as well +* it checks what has been published already, and so won't re-publish/duplicate old tickets (if the inbox gets messy) +* only Esri AGOL Feature Services are supported as endpoints right now; however it would be trivial to store and publish the data in other formats, (e.g., a `csv` or `geojson` in a GitHub repository, a table in CARTO, etc.) For now, this is just an experiment and technology demo; CivicMapper is not affiliated with or contracted by PA OneCall. From 41577a0a0c71abd2ce0d6993ea8c1bf4548cc418 Mon Sep 17 00:00:00 2001 From: Christian Gass Date: Mon, 26 Mar 2018 14:27:42 -0400 Subject: [PATCH 3/3] Create LICENSE --- LICENSE | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..36d9866 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2018 CivicMapper + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE.