Showcasing import_gtfs()

[dhersz]

Hello @tbuckl, @mpadge, @rafapereirabr, @polettif.

I'm creating this discussion to showcase import_gtfs(), and to ask for some feedback from you. @rafapereirabr has already opened an issue on some of the things I'll briefly show you.

First things first, since {gtfsio} is aimed at package authors who maintain GTFS-related packages, I'm trying to maintain the dependencies to a minimum. Currently it only imports {data.table}, to read and manipulate files, {utils} (download.file()) and {zip} (I had some problems using utils::unzip() before, and {zip} is a dependency-free alternative that doesn't rely on any installed files-compactor software).

The package includes a small sample GTFS file built based on and example feed provided by Google.

Basic usage

Just input the path to a GTFS text file. By default, it reads all text files and fields:

library(gtfsio)

data_path <- system.file("extdata/ggl_gtfs.zip", package = "gtfsio")

gtfs <- import_gtfs(data_path)
names(gtfs)
#>  [1] "calendar_dates"  "fare_attributes" "fare_rules"      "feed_info"      
#>  [5] "frequencies"     "levels"          "pathways"        "routes"         
#>  [9] "shapes"          "stop_times"      "stops"           "transfers"      
#> [13] "translations"    "trips"           "agency"          "attributions"   
#> [17] "calendar"
names(gtfs$trips)
#> [1] "route_id"      "service_id"    "trip_id"       "trip_headsign"
#> [5] "block_id"

path also accepts an URL:

gtfs <- import_gtfs("https://raw.githubusercontent.com/r-transit/gtfsio/master/inst/extdata/ggl_gtfs.zip")
names(gtfs)
#>  [1] "calendar_dates"  "fare_attributes" "fare_rules"      "feed_info"      
#>  [5] "frequencies"     "levels"          "pathways"        "routes"         
#>  [9] "shapes"          "stop_times"      "stops"           "transfers"      
#> [13] "translations"    "trips"           "agency"          "attributions"   
#> [17] "calendar"

You can specify which files to read with files (must be a character vector):

gtfs <- import_gtfs(data_path, files = "trips")
names(gtfs)
#> [1] "trips"
names(gtfs$trips)
#> [1] "route_id"      "service_id"    "trip_id"       "trip_headsign"
#> [5] "block_id"

And, similarly, which fields to read with fields (must be a named list, each element being a character vector):

gtfs <- import_gtfs(
  data_path, 
  files = "trips", 
  fields = list(trips = "trip_id")
)
names(gtfs)
#> [1] "trips"
names(gtfs$trips)
#> [1] "trip_id"

fields argument only impacts specified fields, as expected:

gtfs <- import_gtfs(
  data_path, 
  files = c("shapes", "trips"), 
  fields = list(trips = "trip_id"),
)
names(gtfs)
#> [1] "shapes" "trips"
names(gtfs$shapes)
#> [1] "shape_id"            "shape_pt_lat"        "shape_pt_lon"       
#> [4] "shape_pt_sequence"   "shape_dist_traveled"
names(gtfs$trips)
#> [1] "trip_id"

Adjust message output with quiet:

gtfs <- import_gtfs(
  data_path, 
  files = c("shapes", "trips"), 
  fields = list(trips = "trip_id"),
  quiet = FALSE
)
#> Unzipped the following files to /tmp/RtmpvSoT0T/gtfsio:
#>   * shapes.txt
#>   * trips.txt
#> Reading shapes.txt
#> Reading trips.txt

The function returns a gtfs object. I'm still working on the methods, constructors and validators for the class, but you can have a look on the direction I'm going to at R/gtfs.R.

Undocumented fields and files

Undocumented fields and files (i.e. not specified in the official GTFS reference) are always read as characters (i.e. data types are never guessed by data.table::fread()).

Funny enough, the Google example includes an undocumented field in levels.txt (elevation):

gtfs <- import_gtfs(data_path, files = c("levels"))
gtfs$levels
#>    level_id level_index level_name elevation
#> 1:       L0           0     Street         0
#> 2:       L1          -1  Mezzanine        -6
#> 3:       L2          -2 Southbound       -18
#> 4:       L3          -3 Northbound       -24
class(gtfs$levels$elevation)
#> [1] "character"

Raising exceptions

Basic input checking aside, the function might also throw some errors and warnings. To be honest, I refer to them as "exceptions" in the code because I'm not sure if they should be errors or warnings. I wanted to hear from you what you think, since the code is meant to be used by you :). I thought about creating a warnings_as_errors argument, or something along those lines, to convert these warnings to errors, and ultimately leaving the choice of which exception to raise to package maintainers, but that wouldn't be necessary if we all agree on something from the beginning.

If you specify a file that does not exist in files, it will throw a warning.

gtfs <- import_gtfs(data_path, files = c("trips", "oi"))
#> Warning in import_gtfs(data_path, files = c("trips", "oi")): The provided GTFS
#> feed doesn't contain the following text file(s): 'oi'
names(gtfs)
#> [1] "trips"
names(gtfs$trips)
#> [1] "route_id"      "service_id"    "trip_id"       "trip_headsign"
#> [5] "block_id"

Unless none of the files you specified exist. In such case, it will raise the warning and an error (this error is actually not that optional, because zip::unzip() also throws an error when neither file exist - I just created this error to be more explicit on why this error was thrown, because zip::unzip() error message is less clear):

gtfs <- import_gtfs(data_path, files = c("oi"))
#> Warning in import_gtfs(data_path, files = c("oi")): The provided GTFS feed
#> doesn't contain the following text file(s): 'oi'
#> Error in import_gtfs(data_path, files = c("oi")): The provided GTFS feed doesn't contain any of the specified files.

Also, if you specify a file in fields, but not in files (or, alternatively, you meant to read all files, but the one specified in fields doesn't exist), it raises a warning:

gtfs <- import_gtfs(
  data_path, 
  files = c("trips"), 
  fields = list(shapes = "shape_id")
)
#> Warning in import_gtfs(data_path, files = c("trips"), fields = list(shapes =
#> "shape_id")): The following files were specified in 'fields' but either were not
#> specified in 'files' or do no exist: 'shapes'
names(gtfs)
#> [1] "trips"

gtfs <- import_gtfs(data_path,fields = list(nope = "nope_id"))
#> Warning in import_gtfs(data_path, fields = list(nope = "nope_id")): The
#> following files were specified in 'fields' but either were not specified in
#> 'files' or do no exist: 'nope'
names(gtfs)
#>  [1] "calendar_dates"  "fare_attributes" "fare_rules"      "feed_info"      
#>  [5] "frequencies"     "levels"          "pathways"        "routes"         
#>  [9] "shapes"          "stop_times"      "stops"           "transfers"      
#> [13] "translations"    "trips"           "agency"          "attributions"   
#> [17] "calendar"

If you specify some fields in fields that do not exist, it also raises a warning. If none of the specified fields exist, the function returns a NULL data.table:

gtfs <- import_gtfs(
  data_path, 
  files = c("trips"), 
  fields = list(trips = c("trip_id", "ola"))
)
#> Warning in FUN(X[[i]], ...): 'trips' doesn't contain the following field(s):
#> 'ola'
names(gtfs)
#> [1] "trips"
gtfs$trips
#>    trip_id
#> 1:    AWE1
#> 2:    AWE2

gtfs <- import_gtfs(
  data_path, 
  files = c("trips"), 
  fields = list(trips = c("ola"))
)
#> Warning in FUN(X[[i]], ...): 'trips' doesn't contain the following field(s):
#> 'ola'
names(gtfs)
#> [1] "trips"
gtfs$trips
#> Null data.table (0 rows and 0 cols)

So, what do you think about these exceptions? In {gtfstools} I treat all of them as errors, if I recall correctly. Not sure what you prefer.

Final remarks

I haven't covered every detail of the function here. I suggest you taking a quick look at the code if you are interested in knowing every small corner of it :P

I like this feature showcasing discussion format. I think it's a good way to communicate and discuss function/classes/etc behaviour, both expected and observed. What do you think? I plan on creating a similar one to showcase the gtfs class soon.

Cheers!

[rafapereirabr]

Great work, @dhersz . Thanks for the function overview here! As I mentioned on issue #4 , I believe it is more appropriate if gtfsio throws ERRORs when either specified files and fields do not exist in the GTFS feed.

ps. I do like how the function is flexible to read additional files/fields not specivied in the GTFS format though.

[polettif]

Thanks for this thorough showcase @dhersz!

I'm trying to maintain the dependencies to a minimum

Nice. I also had problem with utils::unzip, so I support your decision there.

The function returns a gtfs object. I'm still working on the methods, constructors and validators for the class, but you can have a look on the direction I'm going to at R/gtfs.R.

Wouldn't as_gtfs be a more common naming convention? Anyways, returning a simple gtfs sounds good, other packages can use inherited classes.

Undocumented fields and files (i.e. not specified in the official GTFS reference) are always read as characters (i.e. data types are never guessed by data.table::fread()).

Fine by me, it's straightforward. Could there be a way to provide "custom column specifications"?

Basic input checking aside, the function might also throw some errors and warnings

Like @rafapereirabr, I also support throwing errors as. Warnings are more difficult to handle/interpret especially for a "low level" package meant to be used by other packages (should they be passed on or ignored or...?).

I'll try to have a look at the code soon.

[dhersz]

Thanks for the suggestions @polettif!

Fine by me, it's straightforward. Could there be a way to provide "custom column specifications"?

Good idea. Perhaps a custom_spec argument, that defaults to NULL (and reads extra files/fields as character) or accepts a named list in the following format otherwise (list(extra_file = c(extra_field1 = "character", extra_field2 = "integer"))). How do you like it?

I argue that this argument should only refer to extra files/fields, as import_gtfs() aims to read GTFS according to the standards we have agreed upon, and any conversions should be left to downstream packages (maybe even call the argument extra_spec?). Along this line, I propose that an error is thrown if a "standard" field is specified in this argument.

Warnings are more difficult to handle/interpret especially for a "low level" package meant to be used by other packages (should they be passed on or ignored or...?).

I agree! I'll wait for a bit longer to see if more discussions arise from this thread and adjust the code accordingly.

Wouldn't as_gtfs be a more common naming convention? Anyways, returning a simple gtfs sounds good, other packages can use inherited classes.

I followed the convention Hadley suggests in his Advanced R book for creating constructors, validators and helpers.

Basically this new_gtfs() function would be used by package maintainers to create new objects that inherit from the original gtfs class, and by any methods who may need to create a new objects (check the [ method for the gtfs class). A hypothetical workflow for our packages could be something along the lines of (using gtfstools as an example here):

gtfs = gtfsio::import_gtfs(.......)
gtfs = gtfstools::convert_from_standards(......)
gtfs = gtfsio::new_gtfs(gtfs, subclass = "dt_gtfs", validation_result = validation_object)

I see gtfs() and as_gtfs() functions (who do not yet exist :P) more geared towards end users who may want to convert list objects to gtfs-like objects. I'll work on these functions ASAP.

Having said all that, I don't have much experience with creating S3 objects and basically all my knowledge on the subject comes from reading Advanced R and SO questions. I therefore lack an important practical knowledge on the matter, so if you think I'm getting something wrong here or we would be better following another path let me know!

[dhersz]

Worked on your suggestions and feedback.

As of f4adeb5 import_gtfs() only raises errors upon misspecified arguments.

I also created the extra_spec argument, in line with @polettif suggestion. You can use it to specify how extra fields should be read, and this is only limited to three choices (character, numeric and integer), to keep consistent with the types specified in the standards (i.e. cannot specify Date, factor, or any other types).

Using the same example provided above:

library(gtfsio)

data_path <- system.file("extdata/ggl_gtfs.zip", package = "gtfsio")

gtfs <- import_gtfs(data_path, files = c("levels"))

gtfs$levels
#>    level_id level_index level_name elevation
#> 1:       L0           0     Street         0
#> 2:       L1          -1  Mezzanine        -6
#> 3:       L2          -2 Southbound       -18
#> 4:       L3          -3 Northbound       -24
class(gtfs$levels$elevation)
#> [1] "character"

gtfs <- import_gtfs(
  data_path, 
  files = "levels", 
  extra_spec = list(levels = c(elevation = "integer"))
)

gtfs$levels
#>    level_id level_index level_name elevation
#> 1:       L0           0     Street         0
#> 2:       L1          -1  Mezzanine        -6
#> 3:       L2          -2 Southbound       -18
#> 4:       L3          -3 Northbound       -24
class(gtfs$levels$elevation)
#> [1] "integer"

Note that you cannot use this argument to overwrite the standards:

gtfs <- import_gtfs(
  data_path, 
  files = "levels", 
  extra_spec = list(levels = c(level_index = "character"))
)
#> Error in FUN(X[[i]], ...): The following field(s) from the 'levels' file were specified in 'extra_spec' but are already documented in the official GTFS reference: 'level_index'

It also complains if a field is specified in extra_spec, but not in fields:

gtfs <- import_gtfs(
  data_path, 
  files = "levels", 
  fields = list(levels = c("level_index", "level_id")),
  extra_spec = list(levels = c(elevation = "integer"))
)
#> Error in FUN(X[[i]], ...): The following files were specified in 'extra_spec' but either were not specified in 'fields' or do no exist: 'elevation'

How do you like it?

[rafapereirabr]

It looks great, @dhersz !

[dhersz]

New addition:

Upon parsing failures, and when quiet != FALSE, warning messages are "live printed" to the console, which helps debugging. Better shown than explained:

bad_gtfs.zip is a copy of ggl_gtfs.zip in which I have erased a few columns from the last row of stop_times.txt and erased the last column of a few rows in fare_rules.txt. Reading it with quiet = TRUE results in a few warning messages thrown by data.table::fread() which are not so helpful (we don't know which files caused the warning):

library(gtfsio)

gtfs_path <- system.file("extdata/bad_gtfs.zip", package = "gtfsio")

gtfs <- import_gtfs(gtfs_path)
#> Warning in data.table::fread(file.path(tmpdir, file_txt), select =
#> fields_classes): Stopped early on line 7. Expected 5 fields but found
#> 4. Consider fill=TRUE and comment.char=. First discarded non-empty line:
#> <<a,JSV,1,1>>
#> Warning in data.table::fread(file.path(tmpdir, file_txt), select =
#> fields_classes): Discarded single-line footer: <<AWD1,0:06:45,0:06:45,S6,6>>

That's because all warnings are printed to the console at the same time by the end of the call, and because the warning call doesn't make clear which file was being read at the time (file.path(tmpdir, file_txt) doesn't say much). With quiet = FALSE the output is much clearer.

gtfs <- import_gtfs(gtfs_path, quiet = FALSE)
#> Unzipped the following files to /tmp/Rtmpymeq69/gtfsio:
#>   * calendar_dates.txt
#>   * fare_attributes.txt
#>   * fare_rules.txt
#>   * feed_info.txt
#>   * frequencies.txt
#>   * levels.txt
#>   * pathways.txt
#>   * routes.txt
#>   * shapes.txt
#>   * stop_times.txt
#>   * stops.txt
#>   * transfers.txt
#>   * translations.txt
#>   * trips.txt
#>   * agency.txt
#>   * attributions.txt
#>   * calendar.txt
#> Reading calendar_dates.txt
#> Reading fare_attributes.txt
#> Reading fare_rules.txt
#>   - Stopped early on line 7. Expected 5 fields but found 4. Consider fill=TRUE and comment.char=. First discarded non-empty line: <<a,JSV,1,1>>
#> <<a,JSV,1,1>>
#> Reading feed_info.txt
#> Reading frequencies.txt
#> Reading levels.txt
#> Reading pathways.txt
#> Reading routes.txt
#> Reading shapes.txt
#> Reading stop_times.txt
#>   - Discarded single-line footer: <<AWD1,0:06:45,0:06:45,S6,6>>
#> Reading stops.txt
#> Reading transfers.txt
#> Reading translations.txt
#> Reading trips.txt
#> Reading agency.txt
#> Reading attributions.txt
#> Reading calendar.txt
#> Warning in data.table::fread(file.path(tmpdir, file_txt), select =
#> fields_classes): Stopped early on line 7. Expected 5 fields but found
#> 4. Consider fill=TRUE and comment.char=. First discarded non-empty line:
#> <<a,JSV,1,1>>
#> Warning in data.table::fread(file.path(tmpdir, file_txt), select =
#> fields_classes): Discarded single-line footer: <<AWD1,0:06:45,0:06:45,S6,6>>

Yes, the warnings are still thrown at the end, but I don't see it as a problem tbh.

[polettif]

Looks good, thank you 👍

You can use it to specify how extra fields should be read, and this is only limited to three choices (character, numeric and integer), to keep consistent with the types specified in the standards (i.e. cannot specify Date, factor, or any other types).

I don't think it's necessary to limit it to those three choices. Extra fields are by definition outside of the gtfs spec so it should be possible to read them as their true data types. Or what's the benefit of limiting it?

[dhersz]

I just wrote it like this to keep it "consistent" and aligned to the mindset of keeping conversions inside our packages.

Which types do you think should also be handled inside import_gtfs()? I guess that reading from a custom type would be most useful to date and time fields, but then I see a few points that we'd have to discuss:

• Why would we convert this extra fields (e.g. let's say reading extra_date as a Date column) and not the standard fields (calendar start_date would still be read as an integer, from the standards)?
• How would we use this argument in our packages? Would it be exposed to the users, and let them control the types they want? We'd have to add support to allow reading different types with data.table::fread(). As far as I remember you can specify any type to it, as long as it has an as method (e.g. this is how I read Dates in {gtfstools}, because it doesn't support custom date formats by default).
• Or this argument would remain hidden from the users?

[dhersz]

Ok, in retrospect, this argument only makes sense if we expose it to the final user, after all he/she is the one that knows if and which extra fields and files are present. And, in fact, if we expose it to him/her, we'd have to allow custom types, otherwise the argument is not nearly as useful as it could be (only if you're dealing with integer/numeric fields).

As a matter of personal preference, I'd prefer to keep any custom conversions out of {gtfsio} and inside our packages because:

• It makes import_gtfs() simpler to develop and use.
• I think it's clearer to keep all conversions in a single function (e.g. gtfstools::convert_from_standards() or tidytransit::convert_from_standards()).
• These convertions would not impose a large overhead.
• We would still have to make an arbitrary choice regarding which custom types to support.

Which leads me to question this argument as a whole hahaha. Such is life.

[rafapereirabr]

Hi all. Have we all agreed on this function? I wondering what else needs to be done before we make our first CRAN submission. Any thoughts?

[polettif]

No objections, looks good to me to get this to CRAN

[dhersz]

Cool. I'm working on some final tests and the vignettes, then it should be good for a first CRAN submission.

[dhersz]

v0.1.0 is on CRAN! Keep me posted on any issues you might face or suggestions! Cheers

[mpadge]

Well done @dhersz, great work developing it all so well and so quickly. I'll start integrating asap. Much indebted to you!

[polettif]

Much appreciated @dhersz I'll try it out on tidytransit's development branch.

[polettif]

Just a thought: Could a parameter that lists files that should not be read (like ignore_files or skip_files) param be helpful? I think it's easier to write

import_gtfs("feed.zip", ignore_files = c("shapes", "stop_times"))

than

import_gtfs("feed.zip", files = c("agency", "stops", "routes", "trips", "calendar", "calendar_dates", "transfers", "feed_info"))

[dhersz]

You're right, this argument would definitely make using the function more convenient.

As I said in the latest PR, I've been quite busy lately and probably won't have much time to implement it in the next following weeks or month, so feel free to send a PR in the meantime. I promise I will merge it faster this time lol

[polettif]

No worries, not really an immediately pressing issue. Also, I wasn't quite clear the PR was ready to merge. I'll see what I can do regarding file skipping.

[dhersz]

Hi @polettif. I've made small adjustments to the implementation you had already created in the skip-files branch and merged to master. I'm thinking about submitting a new version (v0.1.1) to CRAN. Any other issues you may want to see resolved before the submission?

[dhersz]

Hi @polettif. I've made small adjustments to the implementation you had already created in the skip-files branch and merged to master. I'm thinking about submitting a new version (v0.1.1) to CRAN. Any other issues you may want to see resolved before the submission?

[polettif]

Thanks 👍 No, I don't have any other issues to resolve currently.