Update lang docs #51

wipfli · 2024-08-28T16:16:53Z

Happy for feedback @nvkelso!

bdon · 2024-08-29T13:59:56Z

basemaps/localization.md


 Protomaps follows OpenStreetMaps's convention where a features's primary name value is is the most common name in the local language(s).

 In practice, this is most often a single name value like:

 - `London` the locality is represented as a simple key, value pair: `name` = `London`

-However, many places have more than one common local languages and Protomaps passes thru OpenStreetMap's convention of concatenating multiple names with a `/` deliminator into a single name value, like:
+However, many places have more than one common local languages and Protomaps passes thru OpenStreetMap's convention of concatenating multiple names with a `/` or `-` deliminator into a single name value, like:


Is this true/relevant anymore now that we require a language to be passed to the style?

It is a bit less relevant than before. Only when the local language and the target language use different scripts you will see these. Example: Map localized to Greek and you are looking at Bozen - Bolzano...

bdon · 2024-08-29T14:00:45Z

basemaps/localization.md


-Protomaps structures localized names using the same `name:{language_code}` formatting as OpenStreetMap.
+If a name from OpenStreetMap contains text in more than one script, then Protomaps breaks up the name into segments. There can be up to 3 segments: `name`, `name2`, and `name3`. Each segment should have a unique script. 


maybe clarify this is the name tag "de-facto primary local name" from OSM

How are the name2 and name3 synthetic properties used in the style?

Is there a downside to overwriting the value of name with (implied) name1?

Does the name splitting only happen when an allow listed delim (/ or -) is observed in the upstream OSM name tag?

maybe clarify this is the name tag "de-facto primary local name" from OSM

OK

How are the name2 and name3 synthetic properties used in the style?

Depends on the target language and the scripts in name2 and name3. For example, if the target language uses a different script from name2, name3, then those appear as second and third line. Other example, if the target language uses the same script as name3, then that appears as first line if the target language is not available.

Is there a downside to overwriting the value of name with (implied) name1?

Not that I know of.

Does the name splitting only happen when an allow listed delim (/ or -) is observed in the upstream OSM name tag?

No, it always happens when an OSM name contains more than one script. There are exceptions for example when you have 5 Arabic Unicode Codepoints, then the Latin letters "AB" and then again 5 Arabic Unicode Codepoints. That could be for example a Latin street letter in an Arabic street label. In that case the text is not segmented. Here is a link to some tests that cover special cases: https://github.com/protomaps/basemaps/blob/60e7d485c7fc6a4b28be525ebc03f6bdd4f20837/tiles/src/test/java/com/protomaps/basemap/names/ScriptSegmenterTest.java#L97-L127

basemaps/localization.md

bdon · 2024-08-29T14:03:51Z

basemaps/localization.md


 ## Positioned glyph font `pmap:pgf:name:*` values

 Protomaps adds additional names for a small set of language scripts, currently just the `Devanagari` script used for Hindi (`name:hi` and `pmap:pgf:name:hi`) and related languages.

 Rendering text in web browsers works for almost all languages and scripts and feels like magic. However, specialized map renderers like MapLibre have to reimplement text rendering and text layout which is complicated when text needs to be curved along linear map features instead of placed only horizontally or vertically. MapLibre normally assumes a one-to-one mapping between glyphs and Unicode codepoints that also exist in MapLibre font files (aka "font stacks") to accomplish the layout for a large but limited number of scripts. Plugins have been developed to extend MapLibre for **right-to-left** scripts like Arabic and Hebrew, and MapLibre has built-in support for **CJK scripts** like Chinese, Japanese, and Korean.

-To facilitate Protomap's support of additional, non-supported scripts in MapLibre (like the Devanagari script used by the Hindi language), Protomaps exports names with "positioned glphys" so MapLibre can use codepoints as indices of positioned glyphs in an additional custom "font stack". While the raw `pmap:pgf:name:*` values look like giberish when inspecting the raw values, they render correctly in MapLibre to the end user.
+To facilitate Protomap's support of additional, non-supported scripts in MapLibre (like the Devanagari script used by the Hindi language), Protomaps exports names with "positioned glphys" so MapLibre can use codepoints as indices of positioned glyphs in an additional custom "font stack". While the raw `pmap:pgf:name:*` values look like gibberish when inspecting the raw values, they render correctly in MapLibre to the end user.


glphys -> glyphs

nvkelso

I worry we're starting to mix up Protomaps the tile schema with Protomaps the map style a bit in these docs.

nvkelso · 2024-08-30T04:52:51Z

basemaps/localization.md

@@ -12,31 +12,81 @@ Protomaps has several localization options for names used in text labels.

 <MaplibreMap/>

-## Default `name` value
+## Local Names


@bdon what's the Title Case versus Sentence case convention elsewhere in docs that we should be following?

No rules right now! I feel Sentence case is more natural, but let's just fix them when we see them.

nvkelso · 2024-08-30T04:53:56Z

basemaps/localization.md


-## Localized `name:*` values


Consider restoring this section heading?

Moved further down and replaced with "Translated Names"

nvkelso · 2024-08-30T04:55:13Z

basemaps/localization.md

+
+If `pmap:script*` is not present on a name, then it means that the name uses the `Latin` script.
+
+Sometimes names might contain text in multiple scripts. In that case `pmap:script` is set to `Mixed`.


Sometimes a name might contain text in multiple scripts. In that case pmap:script is set to Mixed.

Does this happen only if the delim is not present? As a future todo, would we want to split the string until it's no longer Mixed? As it stands it's a little confusing that above we say we do split the various names, but then it's confusing it could possibly still be Mixed.

Updated to

Sometimes segmentation into single scripts fails due for example inconsistent usage of alphabets. In that case pmap:script is set to Mixed.

nvkelso · 2024-08-30T05:00:46Z

basemaps/localization.md

+(pmap:script2 absent)
+```
+
+The OSM name for "Zürich" only uses the Latin script and therefore we use in Protomaps only `name` and leave `pmap:script` away which implies that the script of the `name` is `Latin`. 


The OSM name for "Zürich" only uses the Latin script so we export name and but omit pmap:script (implying the script of the name is Latin).

nvkelso · 2024-08-30T05:01:50Z

basemaps/localization.md

+(pmap:script3 absent)
+```
+
+The OSM name for Hong Kong is "Hong Kong 香港". We break this up into `name` and `name2` in Protomaps. Since the script of `name` is `Latin`, the `pmap:script` tag is omitted. The script of `name2` is `Han` which is encoded in `pmap:script2`.


Should there have been a delim here in "Hong Kong 香港" (either a / or a -)?

Always spell out OpenStreetMap.

Should there have been a delim here in "Hong Kong 香港" (either a / or a -)?

No, it does not have a delimiter. "香港 Hong Kong" is in https://www.openstreetmap.org/node/7414774650

Always spell out OpenStreetMap.

OK

nvkelso · 2024-08-30T05:03:22Z

basemaps/localization.md

 - `name:zh-Hans` = `瑞士`
 - `name:zh-Hant` = `瑞士`
 - _... many other localized values..._

-_NOTE: The Chinese (`zh`) examples above demonstrates how a single language can have multiple writing systems, in this case both simplified Chinese (`zh-Hans`) used in mainland China and tranditional Chinese (`zh-Hant`) used in Taiwan. The value stored in `zh` could be either of those._


Restore the zh note, please.

If we always normalize zh in OSM to the two explicate variants (as is indicated in #51 (comment)), then this can be dropped.

At the moment we only export name:zh-Hant and name:zh-Hans from OpenStreetMap to the tiles. If one or both of these are missing on the OSM feature, but name:zh is available, then we backfill name:zh into name:zh-Hans or name:zh-Hant.

If I remember correctly, you had a technique to say if a name:zh string was written in name:zh-Hans or name:zh-Hant in tilezen. Is that correct? If yes, how did you do it?

nvkelso · 2024-08-30T05:05:29Z

basemaps/localization.md

-
-To help solve this, Protomaps characterizes the scipt used in the default `name` value by adding a `pmap:script` tag.
-
-Values in `pmap:script` follow the [ISO 15924](https://unicode.org/iso15924/iso15924-codes.html) standard codes for the representation of names of scripts and are summarized in the table below.


Please restore this explanation of the ISO language codes as a note below the table.

Added a note that the script names are from Unicode Standard Annex #24: Script Names

nvkelso · 2024-08-30T05:07:57Z

basemaps/localization.md

-| Language |  Native name | `name:*` property | [ISO 639-2 code](https://en.wikipedia.org/wiki/List_of_ISO_639-2_codes) | [ISO_639-1 code](https://en.wikipedia.org/wiki/ISO_639-1) | [ISO_15924 script(s)](https://unicode.org/iso15924/iso15924-codes.html) |
-|--------|-----------------|-----------|-----|----|----|
-| Arabic | اَلْعَرَبِيَّةُ | `name:ar` | ara | ar | `Arabic` |
-| Bengali | বাংলা | `name:bn` | ben | bn | `Bengali` |


What's our longer term plan to support Bengali and Farsi (I think the only ones dropped from this list)?

Bengali to be added in the future as a positioned glyph font

Farsi we call it "Persian"

nvkelso · 2024-08-30T05:08:54Z

basemaps/localization.md

+| ----- | ----- | ----- | ----- |
+| Arabic | اَلْعَرَبِيَّةُ | `name:ar` | `Arabic` |
+| Bulgarian | български | `name:bg` | `Cyrillic` |
+| Chinese (Simplified) | 中文 汉语 | `name:zh-Hans` | `Han` |


Are we exporting zh or differentiating name:zh-Hans and name:zh-Hant?

We are not exporting zh in the tileset, because that is not a useful developer-facing choice of locale. Ideally we normalize zh in the raw data into both zh-Hans and zh-Hant if they are the same.

nvkelso · 2024-08-30T05:09:48Z

basemaps/localization.md

+| Urdu | اردو | `name:ur` | `Arabic` |
+| Vietnamese | Tiếng Việt | `name:vi` | `Latin` |
+
+_*) `Mixed-Japanese` is a custom `pmap:script` value used for labels that contain Hiragana or Katakana mixed with a second or third script. In Japanese, these two scripts often appear in combination with others._


* NOTE: Mixed-Japanese is a custom pmap:script value used for labels that contain Hiragana or Katakana mixed with a second or third script. In Japanese, these two scripts often appear in combination with others.

nvkelso · 2024-08-30T05:13:49Z

basemaps/localization.md


 See more:

 - [Traditional MapLibre Text Rendering](https://oliverwipfli.ch/about-text-rendering-in-maplibre-2023-10-17/)
 - [Devanagari Positioned Glyph Fonts](https://oliverwipfli.ch/devanagari-in-the-protomaps-basemap-with-a-positioned-glyph-font-for-maplibre-2024-06-30/)

-## Styling localized name


While it's ok to take out this placeholder section... we're tilting towards using Protomaps as a platform solution instead of a modular system of tile schema, styles, and data archives. Ideally we offer some tips on how to work with the raw tile data, too?

You're right, I think this section under the basemaps directory concerns the combination of style + tileset.

I think Localization concerns the style, because that is the API surface that developers interact with - generating a GL style for a given language.

I do think there ought to be one section in the Basemaps directory for the schema with no style opinions - probably fleshed out https://docs.protomaps.com/basemaps/layers (issue #1)? That can mention localized name tags, and link out to this Localization page that has more prose?

wipfli · 2024-09-03T12:45:09Z

Thanks for the reviews @nvkelso and @bdon. Please let me know if I missed something.

Update lang docs

796b1fe

bdon reviewed Aug 29, 2024

View reviewed changes

basemaps/localization.md Outdated Show resolved Hide resolved

bdon reviewed Aug 29, 2024

View reviewed changes

nvkelso reviewed Aug 30, 2024

View reviewed changes

Fix order

fa22629

bdon merged commit 6c6875e into protomaps:main Sep 4, 2024
1 check passed

wipfli deleted the lang branch September 4, 2024 12:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update lang docs #51

Update lang docs #51

wipfli commented Aug 28, 2024

bdon Aug 29, 2024

wipfli Aug 29, 2024

bdon Aug 29, 2024

nvkelso Aug 30, 2024

nvkelso Aug 30, 2024

wipfli Sep 3, 2024

bdon Aug 29, 2024

wipfli Sep 3, 2024

nvkelso left a comment

nvkelso Aug 30, 2024

bdon Sep 4, 2024

nvkelso Aug 30, 2024

wipfli Sep 3, 2024

nvkelso Aug 30, 2024 •

edited

Loading

nvkelso Aug 30, 2024

wipfli Sep 3, 2024

nvkelso Aug 30, 2024

wipfli Sep 3, 2024

nvkelso Aug 30, 2024 •

edited

Loading

wipfli Sep 3, 2024

nvkelso Aug 30, 2024

nvkelso Sep 2, 2024

wipfli Sep 3, 2024

nvkelso Aug 30, 2024

wipfli Sep 3, 2024

nvkelso Aug 30, 2024

wipfli Sep 3, 2024

nvkelso Aug 30, 2024

bdon Aug 30, 2024

nvkelso Aug 30, 2024

wipfli Sep 3, 2024

nvkelso Aug 30, 2024

bdon Sep 4, 2024

wipfli commented Sep 3, 2024


		Protomaps structures localized names using the same `name:{language_code}` formatting as OpenStreetMap.
		If a name from OpenStreetMap contains text in more than one script, then Protomaps breaks up the name into segments. There can be up to 3 segments: `name`, `name2`, and `name3`. Each segment should have a unique script.


		If `pmap:script*` is not present on a name, then it means that the name uses the `Latin` script.

		Sometimes names might contain text in multiple scripts. In that case `pmap:script` is set to `Mixed`.


		To help solve this, Protomaps characterizes the scipt used in the default `name` value by adding a `pmap:script` tag.

		Values in `pmap:script` follow the [ISO 15924](https://unicode.org/iso15924/iso15924-codes.html) standard codes for the representation of names of scripts and are summarized in the table below.

Update lang docs #51

Update lang docs #51

Conversation

wipfli commented Aug 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nvkelso left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nvkelso Aug 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nvkelso Aug 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wipfli commented Sep 3, 2024

nvkelso Aug 30, 2024 •

edited

Loading

nvkelso Aug 30, 2024 •

edited

Loading