Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implementing PHDC Scrubber #12

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open

Implementing PHDC Scrubber #12

wants to merge 11 commits into from

Conversation

Gordonei
Copy link
Contributor

@Gordonei Gordonei commented Aug 30, 2020 •
edited
Loading

Overview

As per #10 and #11, this PR readies scrubbing code from PHDC for integration into the COGPN servce.

Guided tour:

  • This commit is pretty much the code, as received from PHDC
  • This commit is the refactoring of comment elements out
  • This commit adds the class that CoGPN will eventually import and call.

Testing

Unit Tests

Tests added in this commit are passing

Ad-hoc Testing

Test Script

Added a test script in this commit. This allows the scrubber to be called directly, e.g.

$ python3 bin/phdc_scrubber_test.py --datadir src/scrubbers/phdc_scrubber/resources \
                                                           --address "Civic Centre, Hertzog Blvd, Foreshore, Cape Town, 8001"

Output looks like this:

2020-09-13 10:24:37,019-PhdcScrubber.__init__ [INFO]: Initialis[ing] PHDC Scrubber...
2020-09-13 10:24:37,019-PhdcScrubber.__init__ [INFO]: Load[ing] Street Types
2020-09-13 10:24:37,019-PhdcScrubber._check_data_file_exists [DEBUG]: '/home/gordon/workspace/cape-of-good-place-names/src/scrubbers/phdc_scrubber/resources/street_types.txt' all set to be read.
2020-09-13 10:24:37,020-PhdcScrubber.__init__ [INFO]: Load[ed] Street Types
2020-09-13 10:24:37,020-PhdcScrubber.__init__ [INFO]: Load[ing] Address Reference
2020-09-13 10:24:37,020-PhdcScrubber._check_data_file_exists [DEBUG]: '/home/gordon/workspace/cape-of-good-place-names/src/scrubbers/phdc_scrubber/resources/street_name_dict.txt' all set to be read.
2020-09-13 10:24:37,029-PhdcScrubber.create_address_reference [DEBUG]: 33312 lines in address dict 1 ('street_name_dict.txt')
2020-09-13 10:24:37,089-PhdcScrubber._check_data_file_exists [DEBUG]: '/home/gordon/workspace/cape-of-good-place-names/src/scrubbers/phdc_scrubber/resources/HERE_StreetNameSuburbDictionary.txt' all set to be read.
2020-09-13 10:24:37,099-PhdcScrubber.create_address_reference [DEBUG]: 44852 lines in address dict 2 ('HERE_StreetNameSuburbDictionary.txt')
2020-09-13 10:24:37,188-PhdcScrubber.__init__ [INFO]: Load[ed] Address Reference
2020-09-13 10:24:37,188-PhdcScrubber.__init__ [INFO]: ...Initialis[ed] PHDC Scrubber!
2020-09-13 10:24:37,188-PhdcScrubber.scrub [DEBUG]: Scrubbing address 'Civic Centre, Hertzog Blvd, Foreshore, Cape Town, 8001'
2020-09-13 10:24:37,188-PhdcScrubber.get_street_info [DEBUG]: 'BLVD' in 'Civic Centre, Hertzog Blvd, Foreshore, Cape Town, 8001'
2020-09-13 10:24:37,188-PhdcScrubber.get_street_info [DEBUG]: potential_street_number='None'
2020-09-13 10:24:37,188-PhdcScrubber.scrub [DEBUG]: street_type='BLVD', street_number='', address_string='CIVIC CENTRE, HERTZOG BLVD, FORESHORE, CAPE TOWN, 8001'
2020-09-13 10:24:37,188-PhdcScrubber.scrub [DEBUG]: postcode=8001
2020-09-13 10:24:37,188-PhdcScrubber.get_matches [DEBUG]: There are '615' possible substrings
2020-09-13 10:24:37,188-PhdcScrubber.get_matches [DEBUG]: There are '0' overlaps with the address dict
2020-09-13 10:24:37,189-PhdcScrubber.get_matches [DEBUG]: There are '2' possible matches
2020-09-13 10:24:37,189-PhdcScrubber.scrub [DEBUG]: Found '2' potential matches in reference dataset
2020-09-13 10:24:37,189-PhdcScrubber.scrub [DEBUG]: Potential Match Counts: 
{'postcode_match': 464, 'suburb_or_town_words_match': 464}
2020-09-13 10:24:37,189-PhdcScrubber.score_match [DEBUG]: streetname match for "HERTZOG"
2020-09-13 10:24:37,189-PhdcScrubber.score_match [DEBUG]: suburb match for "CAPE TOWN"
2020-09-13 10:24:37,189-PhdcScrubber.score_match [DEBUG]: town match for "CAPE TOWN"
2020-09-13 10:24:37,189-PhdcScrubber.score_match [DEBUG]: initial_score=2, final_score=5
2020-09-13 10:24:37,189-PhdcScrubber.score_match [DEBUG]: streetname match for "HERTZOG"
2020-09-13 10:24:37,189-PhdcScrubber.score_match [DEBUG]: suburb match for "CAPE TOWN"
2020-09-13 10:24:37,189-PhdcScrubber.score_match [DEBUG]: town match for "CAPE TOWN"
2020-09-13 10:24:37,189-PhdcScrubber.score_match [DEBUG]: initial_score=1, final_score=4
2020-09-13 10:24:37,189-PhdcScrubber.scrub [DEBUG]: Final matches: 
[(('', 'HERTZOG', 'BLVD', 'CAPE TOWN', 'CAPE TOWN', '8001'), 5),
 (('', 'HERTZOG', 'BLVD', 'CAPE TOWN', 'CAPE TOWN', ''), 4)]
2020-09-13 10:24:37,189-PhdcScrubber.scrub [DEBUG]: new_address='HERTZOG BLVD CAPE TOWN CAPE TOWN 8001', confidence='1.0'
2020-09-13 10:24:37,189-phdc_scrubber_test.<module> [INFO]: Non-Scrubbed Address: 'Civic Centre, Hertzog Blvd, Foreshore, Cape Town, 8001'
2020-09-13 10:24:37,189-phdc_scrubber_test.<module> [INFO]:     Scrubbed Address: 'HERTZOG BLVD CAPE TOWN CAPE TOWN 8001'
2020-09-13 10:24:37,189-phdc_scrubber_test.<module> [INFO]:           Confidence: '100%'

Tested Values

  • Civic Centre, Hertzog Blvd, Foreshore, Cape Town, 8001: 
    2020-09-13 10:27:38,101-PhdcScrubber.scrub [DEBUG]: new_address='HERTZOG BLVD CAPE TOWN CAPE TOWN 8001', confidence='1.0'
2020-09-13 10:27:38,101-phdc_scrubber_test.<module> [INFO]: Non-Scrubbed Address: 'Civic Centre, Hertzog Blvd, Foreshore, Cape Town, 8001'
2020-09-13 10:27:38,101-phdc_scrubber_test.<module> [INFO]:     Scrubbed Address: 'HERTZOG BLVD CAPE TOWN CAPE TOWN 8001'
2020-09-13 10:27:38,101-phdc_scrubber_test.<module> [INFO]:           Confidence: 100%
  • 1 Solomon Tshuku Ave, Site C, Khayelisthsa, Cape Town: 
    2020-09-13 10:32:48,019-PhdcScrubber.scrub [DEBUG]: new_address='1 SOLOMON TSHUKU AVE, SITE C, KHAYELISTHSA, CAPE TOWN', confidence='0.0'
2020-09-13 10:32:48,019-phdc_scrubber_test.<module> [INFO]: Non-Scrubbed Address: '1 Solomon Tshuku Ave, Site C, Khayelisthsa, Cape Town'
2020-09-13 10:32:48,019-phdc_scrubber_test.<module> [INFO]:     Scrubbed Address: '1 SOLOMON TSHUKU AVE, SITE C, KHAYELISTHSA, CAPE TOWN'
2020-09-13 10:32:48,019-phdc_scrubber_test.<module> [INFO]:           Confidence: 0%
  • 40 Stamen St, Retreat, Cape Town 
    2020-09-13 10:39:49,295-PhdcScrubber.scrub [DEBUG]: new_address='40 STAMEN ST RETREAT CAPE TOWN', confidence='0.6'
2020-09-13 10:39:49,295-phdc_scrubber_test.<module> [INFO]: Non-Scrubbed Address: '40 Stamen St, Retreat, Cape Town'
2020-09-13 10:39:49,295-phdc_scrubber_test.<module> [INFO]:     Scrubbed Address: '40 STAMEN ST RETREAT CAPE TOWN'
2020-09-13 10:39:49,295-phdc_scrubber_test.<module> [INFO]:           Confidence: 60%

@Gordonei
Initial commit of scrubbing code received from PHDC.
eccdec9
@Gordonei
Refactoring of PHDC scrubbing code
84f7612 
Attempting to extract common elements that will be used in the COGPN service:
* Loading datafiles (create_street_types_list, create_address_reference)
* Address munging (get_street_info, generate_match_string)
* Matching and scoring matches between string and address (get_matches, score_match)

Otherwise, if run directly, this module should replicate the PHDC scrubbing script's functionality.

**NB** untested, not even run
@Gordonei Gordonei self-assigned this Aug 30, 2020
@Gordonei
Adding unit tests for refactored scurbber functions
2149c20 
Also includes some bug fixes discovered while testing.
@Gordonei
Adding PHDC Scrubber class that will actually be called by CoGPN.
489556b 
And an end-to-end test that makes use of it.
@Gordonei
Bug fix for failed scrub
9cc715b 
Added test case to exercise it
@Gordonei
Making basic scrubber to return a confidence value
093559f 
Hardcoded to a possibly over-confident one
@Gordonei
Adding a test script for the PHDC scrubber
6453c9f
@Gordonei
Bug fixes for PHDC scrubber from using the test script
0535a75 
* Handling special character names in the street types list
* Only taking street numbers if they're in the first half of the address - deals with scenarios where there is a postal code
* More debugging log lines in match scoring
* Only pass in the postal code when it's a postal code match
@Gordonei
Tweaking PHDC test script
6cd239c
@Gordonei Gordonei marked this pull request as ready for review September 13, 2020 08:43
@Gordonei Gordonei mentioned this pull request Sep 13, 2020
Closed
@ColinAnthony
Copy link

ColinAnthony commented Sep 30, 2020 •
edited
Loading

phdcScrubber changes Road name to suburb name if not comma separated

Minor bug as it is likely that successful geocoding of the address will still place it in the correct suburb (provided resolution below suburb level is not required)

@ColinAnthony
Copy link

PhdcScrubber mis-assigning suburb and road type

This bug is more important as it could cause the address to be successfully geocoded to the wrong suburb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@Gordonei Gordonei

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Gordonei @ColinAnthony