Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GCC Puller: Add a generic function to import any layers on GCC to one's schema #1028

Closed
chmnata opened this issue Aug 1, 2024 · 7 comments · Fixed by #1044
Closed

GCC Puller: Add a generic function to import any layers on GCC to one's schema #1028

chmnata opened this issue Aug 1, 2024 · 7 comments · Fixed by #1044
Assignees
@chmnata
Labels
gis

Comments

@chmnata
Copy link
Collaborator

chmnata commented Aug 1, 2024

No description provided.

@chmnata chmnata added the gis label Aug 1, 2024
@chmnata chmnata self-assigned this Aug 1, 2024
chmnata added a commit that referenced this issue Aug 1, 2024
@chmnata
#1028 add generic puller function
16e84d2
@raha1986
Copy link

raha1986 commented Aug 2, 2024

@chmnata Is it ready for @Hanighomi to use?

@raha1986 raha1986 pinned this issue Aug 2, 2024
@chmnata
Copy link
Collaborator Author

chmnata commented Aug 2, 2024

nope! Do you need to pull something other than streetlight?

@raha1986
Copy link

raha1986 commented Aug 2, 2024

@Hanighomi Do you need to dld another layer too?

@Hanighomi
Copy link
Contributor

@chmnata The other layer is gcc_pavement, which has already been extracted and imported into BigData as 'vz_analysis.mbk_pavement_asset' with the same method for mbk_pole. It might be worthwhile to delete it and rerun the process using the code to maintain consistency.

@chmnata
Copy link
Collaborator Author

chmnata commented Aug 2, 2024

Can you check and see if the number of records in vz_analysis.mbk_pavement_asset is the same as the one on gcc?

@Hanighomi
Copy link
Contributor

The diff is 2 records.
image
image

@Nate-Wessel Nate-Wessel unpinned this issue Aug 2, 2024
@gabrielwol
Copy link
Collaborator

@chmnata may be able to easily tackle this issue at the same time: #561

chmnata added a commit that referenced this issue Aug 13, 2024
@chmnata
#1028 update gcc layer puller
072c809
chmnata added a commit that referenced this issue Aug 21, 2024
@chmnata
#1028 update readme
6862e01
chmnata added a commit that referenced this issue Aug 21, 2024
@chmnata
#1028 add requirement.txt
a97acd0
chmnata added a commit that referenced this issue Aug 23, 2024
@chmnata
#1028 update readme
cef4e83
chmnata added a commit that referenced this issue Aug 23, 2024
@chmnata
#1028 reimport functions
85c9de4
chmnata added a commit that referenced this issue Aug 23, 2024
@chmnata
#1028 remove unneccessary packages
0fde059
chmnata added a commit that referenced this issue Aug 23, 2024
@chmnata
#1028 add generic puller function
bb684bf
chmnata added a commit that referenced this issue Aug 23, 2024
@chmnata
#1028 update gcc layer puller
7b59ee2
chmnata added a commit that referenced this issue Aug 23, 2024
@chmnata
#1028 update readme
63f987c
chmnata added a commit that referenced this issue Aug 23, 2024
@chmnata
#1028 add requirement.txt
8b3efad
chmnata added a commit that referenced this issue Aug 23, 2024
@chmnata
#1028 update readme
e8ee420
chmnata added a commit that referenced this issue Aug 23, 2024
@chmnata
#1028 reimport functions
f6c6898
chmnata added a commit that referenced this issue Aug 23, 2024
@chmnata
#1028 remove unneccessary packages
34f3db0
@chmnata chmnata mentioned this issue Aug 23, 2024
Merged
leo-oxu added a commit that referenced this issue Oct 1, 2024
@leo-oxu
#1028 Deduplicate two files for gcc generic puller, combined into one…
6a403a1 
…, now can run gcc.puller_functions.py directly to pull a layer
leo-oxu added a commit that referenced this issue Oct 1, 2024
@leo-oxu
#1028 Bug fixes for Airflow part of get_layer function for huge layer
37b8b8f
leo-oxu added a commit that referenced this issue Oct 3, 2024
@leo-oxu
#1028 Deduplicate get_layer function and combine three insert_data fu…
c89aab5 
…nctions into one
leo-oxu added a commit that referenced this issue Oct 3, 2024
@leo-oxu
#1028 Update comments for gcc puller functions and updae README.md
3c34709
leo-oxu added a commit that referenced this issue Oct 7, 2024
@leo-oxu
#1028 Bug fixes for audited tables
8c4bf96
leo-oxu added a commit that referenced this issue Oct 7, 2024
@leo-oxu
#1028 Deleted outdated layer pulling ipynb file
8fd1985
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@chmnata @gabrielwol
#1028 add generic puller function
8aab80b
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@chmnata @gabrielwol
#1028 update gcc layer puller
22ad30f
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@chmnata @gabrielwol
#1028 update readme
d8d5193
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@chmnata @gabrielwol
#1028 add requirement.txt
5c587e1
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@chmnata @gabrielwol
#1028 update readme
7e3cc05
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@chmnata @gabrielwol
#1028 reimport functions
1d39add
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@chmnata @gabrielwol
#1028 remove unneccessary packages
2e69d5f
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@leo-oxu @gabrielwol
#1028 Deduplicate two files for gcc generic puller, combined into one…
cc1b670 
…, now can run gcc.puller_functions.py directly to pull a layer
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@leo-oxu @gabrielwol
#1028 Bug fixes for Airflow part of get_layer function for huge layer
339e99c
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@leo-oxu @gabrielwol
#1028 Deduplicate get_layer function and combine three insert_data fu…
3e726f8 
…nctions into one
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@leo-oxu @gabrielwol
#1028 Update comments for gcc puller functions and updae README.md
bbb376d
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@leo-oxu @gabrielwol
#1028 Bug fixes for audited tables
93405e4
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@leo-oxu @gabrielwol
#1028 Deleted outdated layer pulling ipynb file
1002f08
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@chmnata @gabrielwol
#1028 add generic puller function
b334c9f
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@chmnata @gabrielwol
#1028 update gcc layer puller
254c57c
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@chmnata @gabrielwol
#1028 update readme
13315c3
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@chmnata @gabrielwol
#1028 add requirement.txt
a884f70
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@chmnata @gabrielwol
#1028 update readme
fc14ee9
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@chmnata @gabrielwol
#1028 reimport functions
96609e4
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@chmnata @gabrielwol
#1028 remove unneccessary packages
b308516
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@leo-oxu @gabrielwol
#1028 Deduplicate two files for gcc generic puller, combined into one…
89c158c 
…, now can run gcc.puller_functions.py directly to pull a layer
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@leo-oxu @gabrielwol
#1028 Bug fixes for Airflow part of get_layer function for huge layer
befa26b
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@leo-oxu @gabrielwol
#1028 Deduplicate get_layer function and combine three insert_data fu…
3c1049e 
…nctions into one
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@leo-oxu @gabrielwol
#1028 Update comments for gcc puller functions and updae README.md
853c2eb
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@leo-oxu @gabrielwol
#1028 Bug fixes for audited tables
a6d306e
gabrielwol pushed a commit that referenced this issue Oct 7, 2024
@leo-oxu @gabrielwol
#1028 Deleted outdated layer pulling ipynb file
6aa6aeb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chmnata chmnata

Labels
gis
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Gcc puller generic CityofToronto/bdit_data-sources
4 participants
@raha1986 @chmnata @gabrielwol @Hanighomi