Add support for PMTiles #1138

giswqs · 2023-09-20T16:58:43Z

PMTiles is a single-file archive format for tiled data. A PMTiles archive can be hosted on a commodity storage platform such as S3, and enables low-cost, zero-maintenance map applications that are "serverless" - free of a custom tile backend or third party provider.

Currently, it is challenging to render large vector datasets with ipyleaflet. PMTiles can be a great option for rendering large vector datasets with ipyleaflet. The folium-pmtiles package supports rendering PMTiles with folium. See below for an example.

This PR tries to add ipyleaflet support for PMTiles. However, I have very limited JavaScript knowledge. I need your help. Thanks.

@martinRenou @davidbrochart @jtmiclat @bdon

protomaps/PMTiles#209
#1134

import folium
from folium.elements import JSCSSMixin
from folium.map import Layer
from jinja2 import Template

class PMTilesMapLibreLayer(JSCSSMixin, Layer):
    """Based of
    https://github.com/python-visualization/folium/blob/56d3665fdc9e7280eae1df1262450e53ec4f5a60/folium/plugins/vectorgrid_protobuf.py
    """

    _template = Template(
        """
            {% macro script(this, kwargs) -%}
            let protocol = new pmtiles.Protocol();
            maplibregl.addProtocol("pmtiles", protocol.tile);

           {{ this._parent.get_name() }}.createPane('overlay');
           {{ this._parent.get_name() }}.getPane('overlay').style.zIndex = 650;
           {{ this._parent.get_name() }}.getPane('overlay').style.pointerEvents = 'none';

            var {{ this.get_name() }} = L.maplibreGL({
            pane: 'overlay',
            style: {{ this.style|tojson}}
            }).addTo({{ this._parent.get_name() }});

            {%- endmacro %}
            """
    )
    default_css = [
        ("maplibre_css", "https://unpkg.com/maplibre-gl@2.2.1/dist/maplibre-gl.css")
    ]

    default_js = [
        ("pmtiles", "https://unpkg.com/pmtiles@2.5.0/dist/index.js"),
        ("maplibre-lib", "https://unpkg.com/maplibre-gl@2.2.1/dist/maplibre-gl.js"),
        (
            "maplibre-leaflet",
            "https://unpkg.com/@maplibre/maplibre-gl-leaflet@0.0.17/leaflet-maplibre-gl.js",
        ),
    ]

    def __init__(self, url, layer_name=None, style=None, **kwargs):
        self.layer_name = layer_name if layer_name else "PMTilesVector"

        super().__init__(name=self.layer_name, **kwargs)

        self.url = url
        self._name = "PMTilesVector"

        if style is not None:
            self.style = style
        else:
            self.style = {}


m = folium.Map(location=[43.7798, 11.24148], zoom_start=13)
pmtiles_url = "https://open.gishub.org/data/pmtiles/protomaps_firenze.pmtiles"
pmtiles_layer = PMTilesMapLibreLayer(
    "folium_layer_name",
    overlay=True,
    style={
        "version": 8,
        "sources": {
            "example_source": {
                "type": "vector",
                "url": "pmtiles://" + pmtiles_url,
                "attribution": '<a href="https://protomaps.com">Protomaps</a> © <a href="https://openstreetmap.org/copyright">OpenStreetMap</a>',
            }
        },
        "layers": [
            {
                "id": "buildings",
                "source": "example_source",
                "source-layer": "landuse",
                "type": "fill",
                "paint": {"fill-color": "steelblue"},
            },
            {
                "id": "roads",
                "source": "example_source",
                "source-layer": "roads",
                "type": "line",
                "paint": {"line-color": "black"},
            },
        ],
    },
)
m.add_child(pmtiles_layer)
folium.LayerControl().add_to(m)
m

giswqs · 2023-09-23T16:51:05Z

Another JS library that can be used for this: https://github.com/protomaps/protomaps-leaflet

bdon · 2023-09-24T00:47:08Z

js/src/leaflet.js

@@ -16,6 +16,7 @@ require('leaflet-fullscreen');
 require('leaflet-transform');
 require('leaflet.awesome-markers');
 require('leaflet-search');
+require('protomaps-leaflet');


for general purpose vector tile display, maplibre is better than protomaps-leaflet, is it possible to use that instead?

I used maplibre in the initial commit, but it failed. I don't how to fix it, so I changed to back to protomaps-leaflet. See
92586cd#diff-ab455b4e55f1402a4baec5dd7dd302a20e715e7b3897e82cc2e4b6ac557baba8

For references on how styling vector tiles works with ipyleaflet. https://ipyleaflet.readthedocs.io/en/latest/layers/vector_tile.html

i ended up using protomaps-leaflet's json_style function to convert mapbox style specs to protomaps style

jtmiclat@c101c87

I found trying to use maplibre is super hacky with overlays and for the case of ipyleaflet adds a large js dependency on build time for an optional feature.

the json_style only parses a small subset of mapbox style specs. I don't have the time or resources to bring that to a complete state (the spec is very complex - look at https://github.com/openlayers/ol-mapbox-style for a comparison), which is why I would rather remove the feature instead of confusing developers as to why their JSON styles don't work. I may re-evaluate this, but in general the state of protomaps-leaflet is frozen as the majority of use cases have moved to MapLibre. This could change if a company or institution wants to sponsor the continued development of the protomaps-leaflet engine, a good use case may be these notebooks.

jtmiclat · 2023-09-26T14:00:33Z

Got a working protomaps-leaflet in the following commit: cafc323

But facing the same issue that there isn't a simple way styling it

giswqs · 2023-09-26T15:25:42Z

Amazing work! That's one big step forward. I am excited

jtmiclat · 2023-09-27T04:17:06Z

@giswqs Pushed a working version with styling to https://github.com/jtmiclat/ipyleaflet/tree/pmtiles. Feel free to merge that branch here. I learned that protomaps-leaflet had a function to convert simple mapbox styles to protomap styles. Might propagate that change to folium-pmtiles

Example usage:

from ipyleaflet import Map, PMTilesLayer

m = Map(center=[43.7798, 11.24148], zoom=13)
vl = PMTilesLayer(url="https://pmtiles.jtmiclat.me/protomaps(vector)ODbL_firenze.pmtiles", 
    style={
        "layers": [
            {
                "id": "landuse",
                "source": "example_source",
                "source-layer": "landuse",
                "type": "fill",
                "paint": {"fill-color": "black"},
            },
            {
                "id": "roads",
                "source": "example_source",
                "source-layer": "roads",
                "type": "line",
                "paint": {"line-color": "steelblue"},
            },
        ],
    })
m.add_layer(vl)
m

Based on protomaps/protomaps-leaflet#112 this function will be removed in the future though!

giswqs · 2023-10-15T02:52:25Z

@jtmiclat Sorry for the delay! I have incorporated your code into this PR. Thank you very much for your help with this.

@martinRenou The new feature allows ipyleaflet to visualize large vector datasets. It will greatly benefit the geospatial community. The unit tests have all passed. Please review it when you have time.

giswqs · 2023-10-15T04:13:24Z

Just added a notebook example for visualizing a 1.1 GB PMTiles.

from ipyleaflet import Map, basemaps, PMTilesLayer

m = Map(center=[52.963529, 4.776306], zoom=7, basemap=basemaps.CartoDB.DarkMatter, scroll_wheel_zoom=True)
m.layout.height = '600px'

vl = PMTilesLayer(url="https://storage.googleapis.com/ahp-research/overture/pmtiles/overture.pmtiles", 
    style = {
        "layers": [
            {
                "id": "admins",
                "source": "example_source",
                "source-layer": "admins",
                "type": "fill",
                "paint": {"fill-color": "#BDD3C7", "fill-opacity": 0.1},
            },
            {
                "id": "buildings",
                "source": "example_source",
                "source-layer": "buildings",
                "type": "fill",
                "paint": {"fill-color": "#FFFFB3", "fill-opacity": 0.5},
            },
            {
                "id": "places",
                "source": "example_source",
                "source-layer": "places",
                "type": "fill",
                "paint": {"fill-color": "#BEBADA", "fill-opacity": 0.5},
            },
            {
                "id": "roads",
                "source": "example_source",
                "source-layer": "roads",
                "type": "line",
                "paint": {"line-color": "#FB8072"},
            },
        ],
    })
m.add(vl)
m

Peek.2023-10-15.00-10.mp4

giswqs · 2023-10-26T11:44:23Z

Can one of the maintainers review and merge this PR?

giswqs · 2023-11-23T12:03:45Z

It would be great if this PR can be included in the next release

martinRenou

Thanks!

giswqs added 4 commits September 20, 2023 12:48

Add support for PMTiles

92586cd

Fix yarnpkg error

a34d3bf

Fix package.json

6261995

Remove maplibre-gl-leaflet

d1370ca

giswqs mentioned this pull request Sep 22, 2023

Add additional parameters for folium layer jtmiclat/folium-pmtiles#5

Merged

giswqs added 4 commits September 23, 2023 16:10

Update package

974af2b

Fix linter issue

93048f4

Fix linter issue

2070e94

Change layer view

ba3cd8d

bdon reviewed Sep 24, 2023

View reviewed changes

giswqs mentioned this pull request Sep 26, 2023

Deno Userland Jupyter Messaging library: comms denoland/deno#20592

Open

Incorporate jtmiclat code

abf9805

Add PMTiles notebook

b0aa5d7

giswqs mentioned this pull request Oct 16, 2023

Add ipyleaflet support for PMTiles opengeos/leafmap#575

Merged

martinRenou approved these changes Nov 23, 2023

View reviewed changes

martinRenou merged commit 6bd0cac into jupyter-widgets:master Nov 23, 2023
3 checks passed

bertcoerver mentioned this pull request Jan 4, 2024

PMtiles not showing on map #1161

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for PMTiles #1138

Add support for PMTiles #1138

giswqs commented Sep 20, 2023 •

edited

Loading

giswqs commented Sep 23, 2023

bdon Sep 24, 2023

giswqs Sep 24, 2023

jtmiclat Sep 25, 2023

jtmiclat Sep 27, 2023 •

edited

Loading

bdon Sep 28, 2023

jtmiclat commented Sep 26, 2023 •

edited

Loading

giswqs commented Sep 26, 2023

jtmiclat commented Sep 27, 2023 •

edited

Loading

giswqs commented Oct 15, 2023

giswqs commented Oct 15, 2023

giswqs commented Oct 26, 2023

giswqs commented Nov 23, 2023

martinRenou left a comment

Add support for PMTiles #1138

Add support for PMTiles #1138

Conversation

giswqs commented Sep 20, 2023 • edited Loading

giswqs commented Sep 23, 2023

bdon Sep 24, 2023

Choose a reason for hiding this comment

giswqs Sep 24, 2023

Choose a reason for hiding this comment

jtmiclat Sep 25, 2023

Choose a reason for hiding this comment

jtmiclat Sep 27, 2023 • edited Loading

Choose a reason for hiding this comment

bdon Sep 28, 2023

Choose a reason for hiding this comment

jtmiclat commented Sep 26, 2023 • edited Loading

giswqs commented Sep 26, 2023

jtmiclat commented Sep 27, 2023 • edited Loading

giswqs commented Oct 15, 2023

giswqs commented Oct 15, 2023

giswqs commented Oct 26, 2023

giswqs commented Nov 23, 2023

martinRenou left a comment

Choose a reason for hiding this comment

giswqs commented Sep 20, 2023 •

edited

Loading

jtmiclat Sep 27, 2023 •

edited

Loading

jtmiclat commented Sep 26, 2023 •

edited

Loading

jtmiclat commented Sep 27, 2023 •

edited

Loading