MapX public API to feed third party app, such as a dedicated search engine

[fxi]

MapX exposes an express API for its own needs. It's not designed to be used from third-party applications, even if it's open and no restriction are applied.

We need to improve this.

Example of possible external usage :

• Aggregator of metadata
• Specialized search engine
• Mapping application
• Dashboards
• GIS
• Scripts for analysis, prediction or other research purpose

Examples of concrete resources request:

• Broad usage

  • Full text search : specific language, any text, returns relevant projects, source and view short metadata as a JSON tree

• Specific usage

  • Get list of available projects : all public project id
  • Get project metadata : for a project id, get project metadata
  • List of sources : for a project, list all source id
  • Get source : for a source id, get source metadata or summary
  • List of views : for a project id, list all views id
  • Get a view : for a view id, list view or view's metadata
  • Get a protobuf/geojson/raster tile
    ...

Some usages are more resources intensive and/or more prone to security issues, but we can list of fundamental features that are currently missing from our stack :

• Authentication. Token based authentication or similar, based on MapX roles. Most MapX services already use encrypted tokens, but it's linked to a specific user, not a third party application. The token holds the validity range, the user ID and such. We need a more general approach.
• Usage monitoring. We need to handle usage to prevent overuse. "Tragedy of common" issues. Some routes in our API could deliver large amount of data.
• Documentation. A public API should be properly documented.

Some questions are unclear :

• Is express the best choice or do we need to migrate towards some dedicated framework ?
• Do we need to instantiate and maintain two separate entities ?
• What alternative to we have, without being locked in or restricted in action ?
• Is proper authentication needed or IP monitoring could be sufficient ?

For me, we need to:

• Keep express and all current routes,
• Add token support generated from MapX interface,
• Add middleware to authenticate request tokens and monitor usage,
• Write proper documentation.

Related tools: jsdoc, swagger, graphql, express, koa, fastAPI, ...

[trepmag]

Hello, indeed, express should be kept for the "public API":

• it can serve the task completely and is already integrated (having a separate framework and even worse, using another language, would requires to extend technical expertise)
• documentation can be done using some available swagger tools integration

Regarding authentication: this is required to differentiate some sort of throttling depending if the user is authenticated or not; for example we need to apply a rate throttling for anonymous that is different for authenticated users.

Also, if we need do scope throttling where we restrict access on given resources depending on some user roles then authentication is required.

In any case, it appears that at least some rate throttling should be applied. And, if the initial purpose of this task is to serve public data then using authentication might not be required at this stage.

Note, rate throttling might be applied at proxy level (traefik)...

Concerning the name, perhaps that 'public API' isn't the best choice since in the future we might need to serve non-public data?!?... Perhaps we can use simply 'web-service' to differentiate it with the API for the UI; it's just a proposition...

[fxi]

MapX, draft of an "easy views search tool"

A search tool that returns a list of items that summarize matched views. A "view" is simply the name MapX gives to the organised document used to describe how to display information. It could contain the configuration for displaying vector layer on a map, fetching remote raster tiles, communicating with WMS server, store tables, text, or basically, anything, but well organised

EXAMPLE of returned values:

{
  n_rows : 1200,
    page : 1, 
    pages_total : 12,
    host : "https://app.mapx.org",
    rows : [
      {idView:"mx-1213-124",idProject:"MX-123-123",source:{},"title":{"en":"test"},"querystring":"?project=MX-12-12"}.
      <...>
    ]
}

EXAMPLE of request:

  language = "english",
  text = "Extractive of coltan", 
  date = "before:2017",
  region = "Africa or Asia"

EXAMPLE of Interpretation:

• language:

  • Input as text
  • Resolve as ISO 2 letters language code [English -> en]
  • Used in further steps

• text:

  • Input as text
  • Resolve as tsvector for fulltext search [Extractive of coltan -> {"coltan":1000, "extractive":2,"of":0}]
  • Compares to
    • titles,
    • abstracts,
    • source metadata
      • titles
      • abstracts
      • attributes labels

• date:

  • Input as text
  • Resolve as Unix epoch range : one day to illimited [before:2017 -> {"min":0, "max";12212323}]
  • Compares to
    • view modified date,
    • source modified_date,
    • source meta
      • relased_at,
      • issued_at,
      • start_at,
      • end_at
    • source data
      • mx_t0,
      • mx_t1

• region

  • Input as
    • lat long (point -> buffer -> polygon),
    • bounding box (polygon),
    • text (geolocate -> point | polygon). [Africa or Asia -> [35,17,-35,50]|[59,23,-12,150]]
  • Resolve as polygon
  • Compares to
    • source bounding box,
    • projects country bounding box,
    • wms bounding box for raster views

• Returns

  • Format JSON or XML
  • Content
    • List of summary objects :
      • idView
      • idProject
      • date modified
      • multi lingual title
      • multi lingual abstract
      • url query
      • (...)
  • host
  • Request date
  • Pagination
  • Metadata

[thomaspiller]

Hi @fxi

I'm just supplementing your previous post with 2-3 pieces of information that came up in previous discussions:

For the text argument, search should not be done in the abstracts or in the attributes labels. What should be parsed is:

• view titles
• source titles
• project titles
• keywords

For the region argument, the order of operations to find the views should be as follows:

1. keywords entered manually in the source: metadata -> text -> geographic keywords
2. highlighted countries in the project configuration
3. source bounding box (with tolerance threshold to avoid edge effects)

* The input should contain the list of countries according to the UN and "Global/World" to ensure the correct spelling of names and to help users in the process.

[fxi]

Thanks !

For the keyword, I forgot to add them in my description.
Both views and sources titles are there.
For the project titles, could you elaborate on the logic to fetch views based on project name, maybe with an example ?

Region:

• In the metadata editor, I will add "geographic keywords / tags " field as it does not exist yet. The iso-mapx converter will have to handle this new field.
• In project config, highlighted countries could be split in two fields :
  • List of countries or regions related to the project, used in search
  • Countries to highlight, used for aesthetics purpose.

[thomaspiller]

Some projects have titles that clearly reflect their content in particular those of WESR (WESR: Climate, WESR: Risk, WESR: Pollution....). I think that parsing the projects title will be especially relevant when a user does a very general search on a topic of interest (risk, climate) or is interested in a group of project (WESR, Chemicals, UNBioLab...).

[PierreLacroix]

Same remark for the geographical location, which is sometimes included in the project's name

[fxi]

What you describe is more a search tools for projects with an integrated geocoding system for titles.

We can imagine the returned data tree having multiple branches. Something like

Input :

{
  "text" : `<text that should match title of views, project or source title or ...>`,
  "date": `<matching date of creation, modification, features or source or ...>`,
   "location": `<location of features, by geocoding> `
}

Output :

{
  n_rows : 1200,
    page : 1, 
    pages_total : 12,
    host : "https://app.mapx.org",
    views : [
      {idView:"mx-1213-124",idProject:"mx_xpoa_saasd",source:{},"title":{"en":"test"},"querystring":"?project=mx_xpoa_saasd"}.
      <...>
    ],
    projects : [
      {idProject:"mx-xpoa-saasd", title:"asda",...,?project=mx_xpoa_saasd"},
      <...>
    ],
    sources : [
     {idSource:"mx-asdad-asdasd",...},
      <...>
   ]
}

This could look like this (non-functional mockup ) in our app :

[PierreLacroix]

Sorry not to be clear enough in my previous comment. There is nothing to geocode on the titles of the project or on the titles of the views. The geographical information will be relevant for users of the Search Engine if we expose the titles