gtfs-structures translation data structure

[kylerchin]

We are attempting to create the standard for https://github.com/rust-transit/gtfs-structure
The original pull request was somewhat abandoned / rejected. rust-transit/gtfs-structure#74

The translation is especially difficult to understand, because it allows arbritary translation either based on the primary key of each table, or by matching field values.

Another challenge is producing a viable data lookup system that can output translated stop names, route names without excess lookups.

CC: @AndrewHC36 @CallMeJai

[CallMeJai]

@AndrewHC36 RE: "Can u elaborate what translation type, agency field, raw translation are?"

Here is my code sketch for reference

enum AgencyField {
  ...
}

...

enum TranslationType {
  Agency { field_name: AgencyField },
  ...
}

struct RawTranslation<T> {
  // confusingly also contains field_name
  translation_type: TranslationType,
  ...
}

RawTranslation is an attempt to make invalid lines of translations.txt as unrepresentable as possible at the data structure level. It's a rewrite of the RawTranslation struct in the original PR which was an intermediary struct for parsing the translations file and creating the translations hashmap. The original struct was all Strings which was closer to the original spec but allowed representation of basically every kind of invalid translation imaginable.

AgencyField is one of a number of enums (one for each table) which limits the field_name of the struct to be one valid for the table in the translation the struct represents. That is, AgencyField can be one of and only one of the field_names in the Agency table. This is again to reduce the space of representative invalid translations.

TranslationType is an enum I got from some of @kylerchin 's initial code, pictured below.

It was initially just an enum of valid table_names, but in my proposed code sketch, the enum has the enumerated valid field names nested within it, so being called TableName would hide that fact. I think a more descriptive name is in order, but I haven't come up with one yet. I am open to suggestions for that as well as the rest of my code sketch.

[kylerchin]

Hello! I've done an analysis of our entire static dataset. Here's what we've found.

https://github.com/catenarytransit/gtfs-dataset-analysis/blob/main/translation_pivot_analysis.csv

I have yet to peel away what this table means, I'll do that in the morning.

So far, there's 7 different types of Chinese covering Cantonese and various traditional character usage in HK and Taiwan, etc.

[andrew-shc]

We should make a summary of all possible types of transaction or lookup patterns for the entire files. Hopefully we can see some patterns arise from that to make a more intuitive design for GTFS translation.
 
Also agree that hierarchy should be implemented to convey the information as intended, unlike the confusingly, although understandable, flat structure in the translations.txt. RawTranslation is good for now as an intermediary data type for internal crate use.

Possible implementation (read the 2nd code block to see the difference):

struct TableTranslation {
    field_name: String,  // provides which column to select from
    translation: String,  // translated text
}

// no more a generalized row matcher since it would have been baked into each table types, though we still need field value which might need to bring back the generalized enum again (TODO)
struct Translations {
    agency: HashMap<LanguageCode, HashMap<RecordId, TableTranslation>>,
    stops: HashMap<LanguageCode, HashMap<RecordSubId, TableTranslation>>,
    routes: HashMap<LanguageCode, HashMap<Value, TableTranslation>>,
    // ...
}

If the types of transactions are dynamic or too complicated:

#[derive(Hash, PartialEq, Eq)]
enum RowMatcher {
    Record(Id),  // translate by each individual records with any value at the field_name
    RecordSub(Id, Id),
    Value(String),  // translate by any record with a certain value at the field_name
}

struct TableTranslation {
    field_name: String,  // provides which column to select from
    translation: String,  // translated text
}

// writing it out actually makes me think that Kyler's original idea to make each table name its own struct field sound reasonable
// table names: all the values allowed as described in the GTFS specification in "table_name"
// LanguageCode: probably use an enum to further limit invalid usage following Jai's general idea
struct Translations {
    agency: HashMap<LanguageCode, HashMap<RowMatcher, TableTranslation>>,
    stops: HashMap<LanguageCode, HashMap<RowMatcher, TableTranslation>>,
    routes: HashMap<LanguageCode, HashMap<RowMatcher, TableTranslation>>,
    // ...
}

1. Since when you translate, you would most likely translate all the texts at once (not translate different parts of route to a different language), so a HashMap with LanguageCode keyed would be better. So when you want to change a language, you can just simple iterate for all translation fields in the HashMap<RowMatcher, TableTranslation> for that language to re-translate the whole thing, or translations.{table}.get({language}).get(RowMatcher::) if there's any record or corresponding field value for a specific text you want to translate (e.g., busses appear or loading new regions in the map).
2. BUT, this row matching uses an enum with some varying memory size, which can cause memory inefficiencies. One way to minimize is to make the Id as small as possible.
3. Regarding the SQL implementation for this, I suppose you can make a separate a table for each combination of table_name, language, and RowMatcher type with ID as the corresponding RowMatcher type
   3.1. Since there are limited table_name and RowMatcher type, we can be certain of how many tables there will be. And so far, the agencies we have checked have limited languages, so that means the combination should not be too large.

@kylerchin @CallMeJai what do you guys think with a language centered approach using enum as the key for accessing row? (I think this is Kyler's original idea too lol)

[CallMeJai]

I think keying by language makes sense since we're doing translations.

1. Could you explain why TableTranslation includes field_name? If I'm reading the GTFS translations spec right, field_name needs to be part of the hashmap key, not the final returned value. I can't imagine a way to guarantee that, for example, field_value translation works without lookup by field_name.

2. I'm not sure how this would swing performance-wise but we could have two versions of each table in the translations struct: one for field_value and one for record_id/record_sub_id depending on the table. Tables that use record_sub_id seem to require record_sub_id and there's no option to translate just by record_id, so each table will use one of two — not one of three — kinds of translation keys. We could also use this to reduce the number of possibilities in the RowMatcher enum.

[andrew-shc]

1. field_name is in TableTranslation because because it helps to know which field we are translating at that record. Like for example, a stops.txt might have two records, one for the stop itself and the other for the stop sequence. When you match that record, there could be multiple fields (columns) that can be translated, so field_name helps which fields to translate at that specific row for that table. A field_value translation wouldn't work without field_name, since which field (column of that table) would it be attempting to match? Like the stop_name and stop_desc for stops.txt.
2. If you're talking about each table can either have record_id or record_sub_id for its first option, yes I agree with that there would be only two kinds of row accessor (the first option or the matching field value). But, I feel like there's a better way to handle row accessors for the 2nd option: matching field values. Perhaps we can make a RecordTranslations and ValueTranslations, where we can have the flexibility for matching-field-values row accessor (the ValueTranslations) to be more of a direct hashmap: HashMap<(field_name, value, language to translate to), translated value>.

[kylerchin]

Perhaps Other(String) enum variant should be included in tables and field_name

[andrew-shc]

I feel like we should go back to use struct fields for table names. For field names, we can use enum with that way.

[kylerchin]

List of translated tables

[kylerchin]

Field value translations

[kylerchin]

By record translations

[kylerchin]

Regarding language types, perhaps we should keep that as a string? Unless we plan on adding every type of language-script-region-variant as allowed under the GTFS standard.

Another thing that's tricky is Chinese variants.

Possible language options under www org:
https://www.w3.org/International/articles/bcp47/
Pivot table of GTFS analysis:

[andrew-shc]

I like how there's more Japanese than English translations.

I guess we can make it String for now, but if there's ever a need for strict usage validation there are crates that can help us:

• (Dormant) https://github.com/oxigraph/oxilangtag
• (Dead) https://github.com/rust-iso/rust_iso15924
• (More dead) https://github.com/johnstonskj/locale-types

(This was suppose to be sent 2 days ago)

[kylerchin]

Woah! The oxi lang thing looks cool, I actually might do that instead.

[kylerchin]

The disadvantage of doing the struct method could be that it's harder to program with maybe, because each table needs to be accessed individually. Actually, as long as they all have the same value , it's easy with function calls. The advantage is that it saves memory, because you don't need a million copies of the table name. CC: @AndrewHC36 2024년 1월 17일 (수) 16:13, Andrew Shen ***@***.***>님이 작성:

…

I feel like we should go back to use struct fields for table names. For field names, we can use enum with that way. — Reply to this email directly, view it on GitHub <#2 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZQTZS4LU5U3HJPE72TXDDYPBSK3AVCNFSM6AAAAABB2OUCYGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DCNRTGIZDI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[andrew-shc]

We now emailing on github discussion?

I originally meant struct fields for each tables, but we can make it a method to make it easier for any future changes to the library or GTFS specification.

[kylerchin]

Could you explain your previous idea and this new one in rust pseudo code? It would make it a bit easier for me to grasp.

[andrew-shc]

Actually, let's look at this at a different angle.

Can we just bake the translations? Especially if most transit agency does not include many language. @kylerchin do you want to run a pivot table on the average number of languages included in the translations file in each agencies?

I was originally thinking of using some n-dimensional key mapping to some nested vectors of translated value, and you just apply some constraints to limit what you want access (including just constraining to a specific record/field_value), but this sounds overcomplicated.

[andrew-shc]

Wait my bad. Did you mean struct method as in the method where we use the table names as the fields of the translation struct? I thought you meant struct method as in the associated function of that struct 🤦

struct Translations {
    agency: HashMap<LanguageCode, HashMap<RowMatcher, TableTranslation>>,
    stops: HashMap<LanguageCode, HashMap<RowMatcher, TableTranslation>>,
    routes: HashMap<LanguageCode, HashMap<RowMatcher, TableTranslation>>,
    // ...
}

[CallMeJai]

What is the plan for the API? Is it the same as the original PR i.e. get_xyz_translated(id: &str, language: &str) and xyz.translate(gtfs: &Gtfs, language: &str)?

[kylerchin]

Posted by maintainer of Rust GTFS structures:

Solution 1

let fr_title: Option<&str> = gtfs.translate_field(StopField::Title, &my_stop, Lang::FR);

-> how can we enforce in the type system that we don't give a RouteField?

Solution 2

let fr_title: Option<&str> = my_stop.translate(Lang::FR, StopField::Title);

-> the problem is for this we would need to store the translations in each items, and since there are translatable field on stop times, I think the memory cost will be too big.

Solution 3

let fr_title: Option<&str> = gtfs.translate_stop_title(my_stop, Lang::FR);

Orthogonal decision

Should Lang be an Enum or a raw str

Enum enforce better typing, but might be a pain to maintain with all possible lang?

Return types

Should the translation return Option<&str> or a &str defaulting to the main lang if no translation is available?
I'd prefer an explicit Option, but it might be more difficult to use (maybe we can have different functions to choose this?).
And if we want a fallback to the original field, should we pass it explicitly like

let fr_title: Option<&str> = gtfs.translate_field(StopField::Title, &my_stop, Lang::FR, &my_stop.title);

leading to potential errors, or should we be smart and have a way to get the title from the enum.

Side note

On a side note, maybe we can first focus on finishing the Raw part of the translations, where we just need to map the specification, without thinking about usage, and iterate of a nice API afterward?

Originally posted by @antoine-de in rust-transit/gtfs-structure#74 (comment)

[CallMeJai]

1. Not sure if this is the neatest solution but something with generic fields on traits should work i.e.

trait TranslatesField<T, U> {
  fn translate_field(T, U, Lang) -> Option<&str>;
}

impl TranslatesField<StopField, &Stop> for Gtfs {
  fn translate_field(StopField, &Stop, Lang) -> Option<&str> {
    ...
  }
 }

but honestly I don't see a case for translating individual fields in the API, so this whole thing might be a non-issue.

2. The original pull request uses gtfs.translate inside the my_stop.translate() call. The translation wouldn't have to be stored multiple times if we used that approach.

3. I think this would make for a really messy, verbose API.

Looks like we might have a good solution for language enums with rust-language-tags.

I also prefer an explicit Option over the implicit failure with just &str. I'm not sure about providing default translations since I think that would depend on stuff like the country and therefore the end-user. I think letting them handle the Option is the best case.

Overall, I think thinking about the API earlier rather than later is better. It helps to inform the design for the entire PR. I wrote a comment asking about it where we could centralize discussion.

[kylerchin]

I do think the API should be implemented as soon as possible. Obviously option makes sense.

I'm not sure what to do in terms of how to store the translations and how to access them still.

[antoine-de]

For the client API, we did a simple POC here.

The Idea would be to have:

my_gtfs.translate(stop.as_ref(), gtfs_structures::StopFields::Name, "en")

Note: for the language it's a separate discussion if we want to have raw strings or a better typed alternative.

The asked Field would be an associated type of a trait defined for each translable type, so it would be impossible to ask for a RouteField::short_name for a Stop.

Would that be ok for you ?

[kylerchin]

I am okay with the method stated above! I would much prefer typed translations.

Also maybe an extra field in the GTFS would be helpful

Vec<TranslatableLanguages>

[kylerchin]

This library https://github.com/pyfisch/rust-language-tags/ looks a lot more comprehensive compared to oxilangtag.

[kylerchin]

Well it works!!! Here's some example usage: https://github.com/catenarytransit/gtfs_translation_test

[kylerchin]

The PR for this discussion is now: rust-transit/gtfs-structure#155

[kylerchin]

Send by me: rust-transit/gtfs-structure#155 (comment)

Should we keep the HashMap like this? Or should we use the trait system? or both? What direction should the lookup mechanism take?

I personally would be alright with the current structure and a trait lookup mechanism, providing options would make the library more friendly for developers.

[kylerchin]

I'm stuck, asking for StopTimes is not possible right now

[kylerchin]

A first beta of a hashmap data structure is now ready here: https://github.com/catenarytransit/gtfs-translations

Further impls will be added to the gtfs structures library eventually