Linked Data Support (LD) in pygeoapi

[doublebyte1]

This issue discusses the current LD support in pygeoapi, and whether and how it could be improved.

Current State

Currently pygeoapi supports both a JSON and JSON-LD encoding. The JSON-LD encoding injects @content, @type and @id into the following resources:

• Landing Page and Collections Page:

  pygeoapi/pygeoapi/linked_data.py

  Line 46 in 46eaaf5

  def jsonldify(func: Callable) -> Callable:

• Collection Page:

  pygeoapi/pygeoapi/linked_data.py

  Line 118 in 46eaaf5

  def jsonldify_collection(cls, collection: dict, locale_: str) -> dict:

• Items Page:

  pygeoapi/pygeoapi/linked_data.py

  Line 176 in 46eaaf5

  def geojson2jsonld(cls, data: dict, dataset: str,

• Item Page:

  pygeoapi/pygeoapi/linked_data.py

  Line 269 in 46eaaf5

  def jsonldify_geometry(feature: dict) -> None:

This is an example of an Item Page: https://demo.pygeoapi.io/master/collections/obs/items/371?f=jsonld
The item page also adds a GeoSparql (gsp) endpoint.

Limitations/ Issues

The default schema is https://schema.org/ and the default gsp is 'http://www.opengis.net/ont/geosparql. It is possible to link properties of the features to terms from the vocabulary.

It is possible to add other vocabularies, but not to override the default vocabulary; the way the multiple vocabularies are handled may originate conflicts with unexpected results. As an example, in this pygeoapi config we define a datetime property based on DCAT:

    linked-data:
        context:
            - schema: https://www.w3.org/ns/dcat#
              stn_id: schema:identifier
              datetime:
                "@id": schema:startDate
                "@type": schema:DateTime

This is a snipped of the resulting JSON-LD definition of a feature:

{
  "@context": [
    {
      "schema": "https://schema.org/",
      "gsp": "http://www.opengis.net/ont/geosparql#",
      "type": "@type"
    },
    {
      "schema": "https://www.w3.org/ns/dcat#",
      "stn_id": "schema:identifier",
      "datetime": {
        "@id": "schema:startDate",
        "@type": "schema:DateTime"
      }
    }
  ],
  "type": "schema:Place",
  "id": 371,
  "linked_data": {
    "context": [
      {
        "schema": "https://www.w3.org/ns/dcat#",
        "stn_id": "schema:identifier",
        "datetime": {
          "@id": "schema:startDate",
          "@type": "schema:DateTime"
        }
      }
    ]
  },
  "stn_id": 35,
  "datetime": "2001-10-30T14:24:55Z",
  "value": 89.9,

Within the @context object we have two schemas defined, which means the last one takes precedence. As a result schema:place will be defined according to DCAT, rather than schema.org. Since dcat#Place does not exist, that link is broken. As the schema.org prefix is not used, the webpage will also not be parsed correctly by the Google search engine crawler.

This could be fixed by having different prefixes for each object within the @context. These behaviours should also be more clearly explained in the documentation.

Another objective which has been widely discussed in terms of visibility of SDIs, is to provide structured data to Google which could originate rich search results. This objective is articulated in the pygeoapi documentation. However, after testing the landing page and other resources with Google tools, it states that no items are detected. This usually means that the JSON-LD syntactically correct, but it does not match any of the specific rich result types that Google Search currently supports.

Verbatim JSON-LD injection feature #2171

This PR enables injecting JSON-LD context into both JSON and JSON-LD Features and STAC items:

• It only acts upon those specific resources.
• It does not update the current LD functionality; when it is switched on the current LD functionality is switched off, which means we can only have one approach or the other.
• It affects both the JSON and JSON-LD encodings, which become equal.

This approach works differently than the current approach:

• It supports only one context, which replaces the default one. The vocabularies, as well as the types are all defined in this context.
• It allows defining a sparql fallback endpoint, within the linked_data structure.
• It supports replacing the id field with the url of the item.

As an example, in this snippet that defines a feature a context is injected:

  {
    "@context": [
      "https://ogcincubator.github.io/bblocks-examples/build/annotated/bbr/examples/observation/vectorObservationFeature/context.jsonld"
    ],
    "id": "https://defs-dev.opengis.net/bblocks-pygeoapi/collections/ogc.bbr.examples.observation.vectorObservationFeature/items/vector-obs-1",
    "type": "Feature",
    "geometry": {
      "type": "LineString",
      "coordinates": [
        [
          -111.67183507997295,
          40.056709946862874
        ],
        [
          -111.71,
          40.156709946862875
        ]
      ]
    },

The linked_data structure is defined like this, reflecting all the options:

"linked_data": {
  "inject_verbatim_context": true,
  "replace_id_field": "id",
  "context": [
    "https://ogcincubator.github.io/bblocks-examples/build/annotated/bbr/examples/observation/vectorObservationFeature/context.jsonld"
  ],
  "fallback_sparql_endpoint": "https://defs-hosted.opengis.net/fuseki-hosted/query"
}

And this is the pygeoapi config that originated this result:

linked-data:
  inject_verbatim_context: true
  replace_id_field: id
  context:
  - https://ogcincubator.github.io/bblocks-examples/build/annotated/bbr/examples/observation/vectorObservationFeature/context.jsonld
  fallback_sparql_endpoint: https://defs-hosted.opengis.net/fuseki-hosted/query

This approach seems more powerful than the current one, as it offsets the context to a separate file where everything can be defined, including setting the use of multiple vocabularies. In this way it restricts the injection of JSON-LD to a single tag.

On the downside, it does require the user to create a context file, even in the simple example that they just want to provide info to schema.org. However, this task could be made easier by reusing the existing OGC building blocks, if this is extensively explained in the documentation.

[rob-metalinkage]

"This approach seems more powerful than the current one, as it offsets the context to a separate file where everything can be defined, including setting the use of multiple vocabularies. In this way it restricts the injection of JSON-LD to a single tag.

On the downside, it does require the user to create a context file, even in the simple example that they just want to provide info to schema.org. However, this task could be made easier by reusing the existing OGC building blocks, if this is extensively explained in the documentation."

This is correct. The use of OGC building blocks in not a requirement - but it may be the only practical way of handling complex schemas such as STAC extensions. For generic "simple features" its possible we could extend with some convenience methods that combine a standard Feature context with a property/URL mapping - in much the same way as BuildingBlocks compiles these - IMHO it is still preferable to use the BuildingBlocks as a FAIR oriented publishing infrastructure.

The current "JSON-LD approach" is at best a placeholder - it made sense given the limitations at that point in the required skill (in tools more than people) for JSON-LD.

There is a widespread misunderstanding around JSON-LD implying a particular schema - in fact its a significant problem that any RDF graph can be serialised many different JSON schemas. Its not obvious that JSON-LD-Framing offers a solution easily supported by available tooling.

The reasons why the existing approach should be replaced with the PR is:

1. it only covers a fraction of the JSON content scope (that matches schema.org, which is designed for a different purpose - discovery not explanation or integration)
2. it stops any other LD mappings from working
3. it can support complex objects such as Records and STAC items
4. it is no longer necessary now we can build "proper" JSON-LD contexts for any feature or complex schema
5. the PR means that JSON-LD does not require complex templating, but could still use it for schema flexibility
6. the PR allows the existing solution to be retained if preferred
7. JSON-LD contexts constructed from standardised components allow clients to recognise and cache common content accessible via the links
8. The PR for a client side library allows for use of a configured store to resolve URI links - this is necessary for many reasons, including interim design and testing of resources not yet published at URIs, CORS header issues and the legacy of "#" fragment based URIs which resolve only to HTML or large documents with unpredictable content models.

probably any one of these concerns is a sufficient reason to update.

_Note the existing JSON-LD templating solution still has value when it is necessary to map an internal structure into a different target, but it should still use a comprehensive FAIR JSON-LD context.

Note also the JSON-LD story has further implications around mapping structural semantics to target ontologies with different structural patterns - such as GeoJSON - GeoSPARQL - however these issues are common to all approaches._

It is, unfortunately, necessary to fully understand all these issues from the perspective of the full lifecycle of designing interoperable data, services and supporting infrastructure. It is also necessary to do a lot of research into tool capabilities. These PRs represent years of developing this understanding and consulting with the JSON-LD, JSON-schema and OpenAPI technical leadership around how to avoid the pitfalls of oversimplified approaches.

[rob-metalinkage]

On the behaviour re the (very specific) case of Google search indexing, this is something that really requires a test harness to be established

This could clearly state the requirements and objective tests.

Assumptions may be hostage to evolving technical and business interests.

Content-negotiation-by-profile could be implemented to support different profiles if these are mutually incompatible. This could probably automatically detect google indexing and serve up the
"schema.org indexing view" without compromising the "describe-the-data view" needed to actually use the data.

[webb-ben]

It is possible to add other vocabularies, but not to override the default vocabulary; the way the multiple vocabularies are handled may originate conflicts with unexpected results.

As implemented here the default context is included in a set of context objects. The behavior could always be using the default context OR the configured context.

It is technically possible to remove the default '@context' via the JSON-LD jinja template as described here (as overkill as it may be) and implemented here.

We have a playground to build such a template at https://docs.geoconnex.us/playground/templating

[webb-ben]

Another objective which has been widely discussed in terms of visibility of SDIs, is to provide structured data to Google which could originate rich search results. This objective is articulated in the pygeoapi documentation. However, after testing the landing page and other resources with Google tools,

As far as my understanding with Google Rich Data tests, this is a documented issue in #877 and further expanded in #916 (comment), what constitutes "valid content" does not include certain HTML views produced by OGC API

[webb-ben]

The default schema is schema.org and the default gsp is 'http://www.opengis.net/ont/geosparql. It is possible to link properties of the features to terms from the vocabulary.

It is the intent for geojson2geojsonld to be a relatively naive method. The intent of this method is to provide a core set of modifications to the JSON content to allow for a Jinja template to produce anything more complicated. We could have the geometry transform use the fully expanded URL, but the point of this method is to correctly set the @id, flatten feature properties, and provide valid GeoSPARQL (WKT) and Schema (I have no idea what standard this is) versions of a features geometry.

It supports replacing the id field with the url of the item.

On a related note, replace_id_field seems like a duplicative implementation of uri_field to set feature @id. Perhaps it makes more sense to include this in the linked_data section of the configuration, although that would be a breaking change.