Skip to content

Commit 0ff15d5

Browse files
authored
Merge pull request #3559 from cal-itp/3497-update-agency-information
Update NTD agency information
2 parents 0dfe3b6 + 81fe56d commit 0ff15d5

File tree

11 files changed

+247
-148
lines changed

11 files changed

+247
-148
lines changed

.github/workflows/publish-docs.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ jobs:
2929

3030
- name: Build jupyter book
3131
run: jb build docs --warningiserror --keep-going # set doc to fail on any sphinx warning
32-
- uses: actions/upload-artifact@v2
32+
- uses: actions/upload-artifact@v3
3333
if: always()
3434
with:
3535
name: docs-build

airflow/dags/create_external_tables/ntd_data_products/annual_database_agency_information.yml

Lines changed: 23 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -16,125 +16,88 @@ hive_options:
1616
source_uri_prefix: "annual-database-agency-information/{dt:DATE}/{ts:TIMESTAMP}/{year:INTEGER}/"
1717
schema_fields:
1818
- name: number_of_state_counties
19-
type: FLOAT
20-
mode: NULLABLE
19+
type: NUMERIC
2120
- name: tam_tier
2221
type: STRING
23-
mode: NULLABLE
2422
- name: personal_vehicles
25-
type: FLOAT
26-
mode: NULLABLE
23+
type: NUMERIC
2724
- name: density
2825
type: FLOAT
29-
mode: NULLABLE
3026
- name: uza_name
3127
type: STRING
32-
mode: NULLABLE
3328
- name: tribal_area_name
3429
type: STRING
35-
mode: NULLABLE
3630
- name: service_area_sq_miles
37-
type: FLOAT
38-
mode: NULLABLE
31+
type: NUMERIC
3932
- name: total_voms
40-
type: FLOAT
41-
mode: NULLABLE
33+
type: NUMERIC
4234
- name: city
4335
type: STRING
44-
mode: NULLABLE
4536
- name: fta_recipient_id
46-
type: FLOAT
47-
mode: NULLABLE
37+
type: NUMERIC
4838
- name: region
49-
type: FLOAT
50-
mode: NULLABLE
39+
type: NUMERIC
5140
- name: state_admin_funds_expended
52-
type: FLOAT
53-
mode: NULLABLE
41+
type: NUMERIC
5442
- name: zip_code_ext
55-
type: FLOAT
56-
mode: NULLABLE
43+
type: STRING
5744
- name: zip_code
58-
type: FLOAT
59-
mode: NULLABLE
45+
type: STRING
6046
- name: ueid
6147
type: STRING
62-
mode: NULLABLE
6348
- name: address_line_2
6449
type: STRING
65-
mode: NULLABLE
6650
- name: number_of_counties_with_service
67-
type: FLOAT
68-
mode: NULLABLE
51+
type: NUMERIC
6952
- name: reporter_acronym
7053
type: STRING
71-
mode: NULLABLE
7254
- name: original_due_date
73-
type: INTEGER
74-
mode: NULLABLE
55+
type: STRING
7556
- name: sq_miles
76-
type: FLOAT
77-
mode: NULLABLE
57+
type: NUMERIC
7858
- name: address_line_1
7959
type: STRING
80-
mode: NULLABLE
8160
- name: p_o__box
8261
type: STRING
83-
mode: NULLABLE
8462
- name: fy_end_date
85-
type: INTEGER
86-
mode: NULLABLE
63+
type: STRING
8764
- name: reported_by_ntd_id
8865
type: STRING
89-
mode: NULLABLE
9066
- name: population
91-
type: FLOAT
92-
mode: NULLABLE
67+
type: NUMERIC
9368
- name: reporting_module
9469
type: STRING
95-
mode: NULLABLE
9670
- name: service_area_pop
97-
type: FLOAT
98-
mode: NULLABLE
71+
type: NUMERIC
9972
- name: subrecipient_type
10073
type: STRING
101-
mode: NULLABLE
10274
- name: state
10375
type: STRING
104-
mode: NULLABLE
10576
- name: volunteer_drivers
106-
type: FLOAT
107-
mode: NULLABLE
77+
type: NUMERIC
10878
- name: primary_uza
109-
type: FLOAT
110-
mode: NULLABLE
79+
type: NUMERIC
11180
- name: doing_business_as
11281
type: STRING
113-
mode: NULLABLE
11482
- name: reporter_type
11583
type: STRING
116-
mode: NULLABLE
11784
- name: legacy_ntd_id
11885
type: STRING
119-
mode: NULLABLE
12086
- name: voms_do
121-
type: FLOAT
122-
mode: NULLABLE
87+
type: NUMERIC
12388
- name: url
12489
type: STRING
125-
mode: NULLABLE
12690
- name: reported_by_name
12791
type: STRING
128-
mode: NULLABLE
12992
- name: voms_pt
130-
type: FLOAT
131-
mode: NULLABLE
93+
type: NUMERIC
13294
- name: organization_type
13395
type: STRING
134-
mode: NULLABLE
13596
- name: agency_name
13697
type: STRING
137-
mode: NULLABLE
13898
- name: ntd_id
13999
type: STRING
140-
mode: NULLABLE
100+
- name: division_department
101+
type: STRING
102+
- name: state_parent_ntd_id
103+
type: STRING

docs/warehouse/warehouse_starter_kit.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ For a given day:
6565

6666
### Other
6767

68-
- [dim_annual_ntd_agency_information](https://dbt-docs.calitp.org/#!/model/model.calitp_warehouse.dim_annual_database_agency_information)
68+
- [dim_annual_agency_information](https://dbt-docs.calitp.org/#!/model/model.calitp_warehouse.dim_annual_database_agency_information)
6969

7070
- View some of the data produced by the [US Department of Transportation](https://www.transit.dot.gov/ntd) for the National Transit Database.
7171
- Information from 2018-2021 are available.

warehouse/models/docs/_docs_ntd.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,10 @@ Docs for NTD models;
22

33
{% docs ntd_id %}
44
A five-digit identifying number for each agency used in the current NTD system.
5+
FTA assigns each reporter a unique five-digit NTD Identification Number.
6+
The first digit of the NTD ID corresponds to the FTA Region where the reporter is located (e.g., 9#### indicates Region IX).
7+
The code will have a four-to-five digit prefix for any entity submitting the report on behalf of the reporter.
8+
For example, State Departments of Transportation (usually indicated as #R##) submit on behalf of their subrecipients.
59
{% enddocs %}
610

711
{% docs ntd_legacy_id %}
@@ -40,6 +44,7 @@ The state in which the agency is headquartered.
4044
{% enddocs %}
4145

4246
{% docs ntd_primary_uza_code %}
47+
The primary urbanized area served by the transit agency.
4348
UACE Code remains consistent across census years.
4449
{% enddocs %}
4550

warehouse/models/docs/_docs_transit_database.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -200,7 +200,7 @@ are implemented for future schema consistency, but historical data has not yet b
200200

201201
{% docs ntd_agency_info_table %}
202202

203-
DEPRECATED: Please use mart_ntd.dim_annual_ntd_agency_information going forward.
203+
DEPRECATED: Please use mart_ntd.dim_annual_agency_information going forward.
204204

205205
2018 NTD Agency Info Table
206206
Imported 10/6/2021 from fta.gov

warehouse/models/mart/ntd/_mart_ntd.yml

Lines changed: 125 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -102,33 +102,35 @@ x-common-fields:
102102
description: '{{ doc("ntd_xlsx_execution_ts") }}'
103103

104104
models:
105-
- name: dim_annual_ntd_agency_information
105+
- name: dim_annual_agency_information
106106
description: >
107-
Versioned extracts of the NTD Annual Database Agency Information.
107+
Contains basic contact and agency information for each NTD reporter.
108108
109-
The versioning is bitemporal, so records are versioned at the year
109+
The dataset can be found at:
110+
https://www.transit.dot.gov/ntd/data-product/2023-annual-database-agency-information
111+
* For other years, just replace 2023 by the desired year.
110112
111-
and ntd_id level. This means you must join based on
112-
_valid_from/_valid_from
113+
The versioning is bitemporal, so records are versioned at the year, ntd_id, and state_parent_ntd_id level.
114+
This means you must join based on _valid_from/_valid_from to get the records for a given ntd_id and state_parent_ntd_id,
115+
and then choose which year to look up.
113116
114-
to get the records for a given ntd_id, and then choose which year to
115-
116-
look up.
117+
Use _is_current to find the latest version for each set of year, ntd_id, and state_parent_ntd_id.
117118
tests:
118119
- dbt_utils.mutually_exclusive_ranges:
119120
lower_bound_column: _valid_from
120121
upper_bound_column: _valid_to
121-
partition_by: CONCAT(year, '_', ntd_id)
122+
partition_by: CONCAT(year, '_', ntd_id, '_', COALESCE(state_parent_ntd_id, ''))
122123
gaps: required
123124
columns:
124125
- name: key
125126
tests:
126127
- not_null
127128
- unique
128-
- name: year
129+
- <<: *report_year
130+
name: year
129131
tests:
130132
- not_null
131-
- name: ntd_id
133+
- <<: *ntd_id
132134
tests:
133135
- not_null
134136
- name: _valid_from
@@ -138,8 +140,120 @@ models:
138140
tests:
139141
- not_null
140142
- name: _is_current
143+
description: Indicates the latest report version for each year, ntd_id, and state_parent_ntd_id.
141144
tests:
142145
- not_null
146+
- name: state_parent_ntd_id
147+
description: |
148+
Indicates the ID number of the transit agency reporting to the database on behalf of the transit agency.
149+
- name: agency_name
150+
description: |
151+
The agency name is the full legal name of the agency.
152+
If reporting is required under an FTA grant program, this must reflect the legal name of the funding recipient.
153+
- name: doing_business_as
154+
description: The name under which the reporting agency is doing business.
155+
- name: address_line_1
156+
description: First line of the agency's mailing address.
157+
- name: address_line_2
158+
description: Second line of the agency's mailing address (if applicable).
159+
- name: p_o__box
160+
description: The PO Box of the agency (if applicable).
161+
- name: city
162+
description: City of the agency's mailing address.
163+
- name: state
164+
description: State of the agency's mailing address.
165+
- name: zip_code
166+
description: Zip Code of the agency's mailing address.
167+
- name: zip_code_ext
168+
description: Zip Code Extension of the agency's mailing address.
169+
- name: region
170+
description: The FTA region in which the reporter is located.
171+
- name: density
172+
description: The population density of the Primary UZA of the agency, if one exists.
173+
- name: ueid
174+
description: |
175+
The UEID is a number or other identifier used to identify a specific commercial, nonprofit, or Government entity.
176+
This is now reported in place of DUNS number for each unique transit agency reporting to the NTD.
177+
See the U.S. General Services Administration UEID web page for more information.
178+
- name: fta_recipient_id
179+
description: |
180+
The four-digit number assigned to a transit agency for the Federal Transit Administration (FTA) electronic grant making system — TrAMS (Transportation Award Management System).
181+
- name: original_due_date
182+
description: The date on which the 2020 NTD Report was due to FTA.
183+
- name: fy_end_date
184+
description: Calendar selection for the last day of an agency's fiscal year.
185+
- name: number_of_counties_with_service
186+
description: |
187+
States report the total number of counties in the state that are currently served, in whole or in part, by Formula Grants for Rural Areas (§5311)-funded operators.
188+
- name: number_of_state_counties
189+
description: The number of Counties in given State (for State Departments of Transportation).
190+
- *organization_type
191+
- name: personal_vehicles
192+
description: |
193+
Vehicles that are used by the transit provider to transport passengers in revenue service but are owned by private individuals, typically an employee of the agency or a volunteer driver.
194+
- name: population
195+
description: The population of the Primary UZA of the agency, if one exists.
196+
- *primary_uza_code
197+
- *primary_uza_name
198+
- name: reported_by_name
199+
description: The NTD ID of the entity reporting on behalf of another entity.
200+
- name: reported_by_ntd_id
201+
description: |
202+
The entity, usually a State, submitting an NTD report on behalf of another entity, usually a subrecipient of the State.
203+
- name: reporter_acronym
204+
description: The acronym used by the reporting agency.
205+
- name: reporter_type
206+
description: |
207+
Reporter Type will be based on where they operate and the reporting requirements associated with their agency.
208+
Agencies that receive Chapter 53 funds and own, operate, or manage capital assets in public transportation are also required to file an annual report, even if they do not receive §5307 or §5311 funds.
209+
Agencies that do not receive or benefit from FTA funding may elect to submit their data to the NTD as Voluntary Reporter but are still assigned a reporter type.
210+
Current types are:
211+
`Building Reporter`,
212+
`Full Reporter`,
213+
`Group Plan Sponsor`,
214+
`Planning Reporter`,
215+
`Reduced Asset Reporter`,
216+
`Reduced Reporter`,
217+
`Rural Reporter`,
218+
`Separate Service`,
219+
`State Reporter`.
220+
- name: reporting_module
221+
description: |
222+
A general classification that will determine which, if any, FTA formula programs will use the NTD data.
223+
For example, Tribes and Native Villages will have data included in the in the §5311j Tribal Transit Program.
224+
Reporters receiving Chapter 53 funds but not receiving or benefiting from §5307 and §5311
225+
AND not electing to report service data are classified as Asset due to the requirement to report asset inventory data.
226+
These agencies are not presently included in formula program datasets.
227+
- name: service_area_pop
228+
description: |
229+
A measure of access to transit service in terms of population served and area coverage (square miles).
230+
The reporting transit agency determines the service area boundaries and population for most transit services using the definitions contained in the Americans with Disabilities Act of 1990 (ADA),
231+
i.e. a corridor surrounding the routes 3/4 of a mile on either side, or for rail, a series of circles of radius 3/4 mile centered on each station Transit agency reporters are required to submit service area information.
232+
- name: sq_miles
233+
description: The square miles of the Primary UZA of the agency, if one exists.
234+
- *service_area_sq_miles
235+
- name: state_admin_funds_expended
236+
description: |
237+
States report the §5311 revenues they expended as a result of administering the program.
238+
Since the §5311 program operates on a reimbursement basis, revenues expended during the report year will be expended during the same year.
239+
Report the operating revenue expended during the report year from FTA §5311 Formula Grants for Rural Areas funds.
240+
- name: subrecipient_type
241+
description: Reflects the type of Rural Formula Grant funding received by the subrecipient.
242+
- name: tam_tier
243+
description: |
244+
Defines whether the agency is a Tier I agency required to produce their own Transit Asset Management plan (and, in parenparens, on what basis)
245+
or a Tier II operator eligible to be in a group TAM Plan.
246+
N/A indicates that the requirement does not apply.
247+
- name: total_voms
248+
description: |
249+
The Vehicles Operated in Maximum Service ("peak service level") across the entire fiscal year for the given agency.
250+
- name: tribal_area_name
251+
description: The tribal land, determined by US Census data, on which tribes operate.
252+
- name: url
253+
description: Agency's transit website.
254+
- name: volunteer_drivers
255+
description: |
256+
Individuals who drive vehicles in revenue service to transport passengers for the transit provider but are not employees of the transit provider and are not compensated for their labor.
143257
144258
- name: dim_annual_funding_sources
145259
description: >-

0 commit comments

Comments
 (0)