incremental-jsonsystem

proxy microservice that implements SESAM pull protocol on a json source system.

can be used to:

• implement support for sesam's since parameter
• extract entities from a field in response json
• add sesam's _updated field to support continuation
• sorting after _updated field value to support chronological continuation
• limit the number of returned entities
• streaming page by page

Environment Parameters

CONFIG_NAME	DESCRIPTION	IS_REQUIRED	DEFAULT_VALUE
FULL_URL_PATTERN	pattern for the full run url. see URL_PATTERN below.	yes	n/a
UPDATED_URL_PATTERN	pattern for the incremental run url. see URL_PATTERN below.	yes	n/a
CONFIG	json with optional headers and optional oauth2 fields which are also json.	yes	n/a
AUTHENTICATION	OAUTH2 for oauth2 support. Any other value for no auth	no	n/a
LOG_LEVEL	Log level. One of "DEBUG	INFO	WARNING
UPDATED_PROPERTY	(DEPRECATED) use corresponding query parameter instead.	no	n/a
OFFSET_BIGGER_AND_EQUAL	(DEPRECATED) use corresponding query parameter instead.	no	n/a

Query Parameters

Query parameters that the service supports can be divided into three groups as described below.

Query parameters can be specified in the FULL_URL_PATTERN/UPDATED_URL_PATTERN or in the url property of the pipe. If specified in both, the latter will be given precedence.

1. query parameters after SESAM pull protocol

These parameters are supported due to SESAM pull protocol. Note that these parameters are sent implicitly by the SESAM depending on the continuation support setings in the pipe config.

CONFIG_NAME	DESCRIPTION	IS_REQUIRED	DEFAULT_VALUE
since	see SESAM pull protocol	no	n/a
limit	see SESAM pull protocol	no	n/a

2. query parameters after proxy service:

There are the parameters that effect the way this microservice works. They are all prefixed with "ms_".

CONFIG_NAME	DESCRIPTION	IS_REQUIRED	DEFAULT_VALUE
ms_since_param_at_src	the name of the query parameter at the source system that corresponds to sesam's since parameter	no	n/a
ms_limit_param_at_src	the name of the query parameter at the source system that corresponds to sesam's limit parameter	no	n/a
ms_updated_property	the name of the field that will be read into _updated field for each entity. Supports dot notation.	no	n/a
ms_use_currenttime_as_updated	sometimes the source does not expose updated date. In such cases, this flag can be set to true to use currenttimestamp as the _updated value.	no	n/a
ms_data_property	the path to the field from which the entities will be read. Supports dot notation.	no	n/a
ms_do_sort	flag to get output sorted after _updated field. Values: true/false. Applies to the page that is being retrieved only, hence upon pagination sorting is guaranteed only if the source delivers entities sorted	no	false
ms_offset_bigger_and_equal	set to true to get the entities that are greater than the offset value instead of greater-than-or-equals. The source systems behaviour should be taken into account here. Works for offset values of type integer only. Values: true/false	no	false
ms_pagenum_param_at_src	if pagination is applicable,the name of the query parameter for the page number. Required to activate pagination along with the "page number parameter" so that succeeding pages would be returned as well	no	n/a

3. query parameters after the Source system

There are the parameters that are passed over to the source system.

An example of system config:

system:

{
  "_id": "incremental-jsonsystem_id",
  "type": "system:microservice",
  "connect_timeout": 60,
  "docker": {
    "environment": {
      "URL_PATTERN": "http://external-host:8888/__PATH__?offset=__since__&limit=__limit__",
      "AUTHENTICATION": "oauth2",
      "CONFIG": {
        "oauth2": {
          "client_id": "id",
          "client_secret": "secret",
          "resource": "resource",
          "token_url": "url"
        },
        "headers": {
          "x-api-version": "1.0"
        },
      "UPDATED_PROPERTY": "sequenceNumber"
    },
    "image": "sesamcommunity/incremental-jsonsystem:x.y.z",
    "port": 5000
  },
  "read_timeout": 7200,
}

system and pipe

{
  "_id": "incremental-jsonsystem_id",
  "type": "system:microservice",
  "connect_timeout": 60,
  "docker": {
    "environment": {
      "URL_PATTERN": "http://external-host:8888/__PATH__=ms_pagenum_param_at_src=pagenum&pagenum=1&pagesize=100",
      "AUTHENTICATION": "oauth2",
      "CONFIG": {
        "oauth2": {
          "client_id": "id",
          "client_secret": "secret",
          "resource": "resource",
          "token_url": "url"
        },
        "headers": {
          "x-api-version": "1.0"
        }
    },
    "image": "sesamcommunity/incremental-jsonsystem:x.y.z",
    "port": 5000
  },
  "read_timeout": 7200,
}

{
  "_id": "my-pipe-id",
  "type": "pipe",
  "source": {
    "type": "json",
        "system": "incremental-jsonsystem_id",
        "is_chronological": true,
        "is_since_comparable": true,
        "supports_since": true,
        "url": "mypath/yourpath/ourpath?ms_data_property=property1.property2&ms_since_param_at_src=updated_since&ms_updated_property=updated_at&ms_do_sort=true&pagesize=50"
  },
  "transform":  [...]
}

URL_PATTERN

A valid URL_PATTERN is any template string that will reveal a url after optionally being exposed to several predefined replacements.

It should suffice to use only __path__ replacement in most cases. In case not, you can use other replacements described here.

Replacements :

• __path__: Will be replaced with the path int the url field of the input pipe
• __since__: Will be replaced with the since value that is sent by SESAM upon pipe execution. (applicable to pipes with continuation support)
• __limit__: Will be replaced with the since value that is sent by SESAM upon pipe execution.

examples

• ENV(my-base-url)/__path__
• ENV(my-base-url)/__path___/__since__
• ENV(my-base-url)/__path__?since=__since__
• ENV(my-base-url)/__path__?since=__since__&pagesize=__limit__