Merge pull request #255 from SoftwareAG/develop

Bugfix release 4.5.1

ARCHITECTURE.md

@@ -1,11 +1,12 @@
 # Architecture
+
 The solution is composed of two major components:
 
-* A **microservice** - exposes REST endpoints, provides a generic connector interface which can be used by broker clients to
+- A **microservice** - exposes REST endpoints, provides a generic connector interface which can be used by broker clients to
   connect to a message broker, a generic data mapper, a comprehensive expression language for data mapping and the
   [Cumulocity Microservice SDK](https://cumulocity.com/guides/microservice-sdk/introduction/) to connect to Cumulocity. It also supports multi tenancy.
 
-* A **frontend (plugin)** - uses the exposed endpoints of the microservice to configure a broker connection & to perform
+- A **frontend (plugin)** - uses the exposed endpoints of the microservice to configure a broker connection & to perform
   graphical data mappings within the Cumumlocity IoT UI.
 
 The architecture of the components consists of the following components:
@@ -16,22 +17,23 @@ The architecture of the components consists of the following components:
 <br/>
 The light blue components are part of this project which are:
 
-* three default connectors for..
-    * **MQTT client** - using [hivemq-mqtt-client](https://github.com/hivemq/hivemq-mqtt-client) to connect and subscribe to MQTT brokers
-    * **MQTT Service client** - using hivemq-mqtt-client to connect to MQTT Service
-    * **Kafka connector** - to connect to Kafka brokers
-* **Data mapper** - handling of received messages via connector and mapping them to a target data format for Cumulocity IoT.
+- three default connectors for..
+  - **MQTT client** - using [hivemq-mqtt-client](https://github.com/hivemq/hivemq-mqtt-client) to connect and subscribe to MQTT brokers
+  - **MQTT Service client** - using hivemq-mqtt-client to connect to MQTT Service
+  - **Kafka connector** - to connect to Kafka brokers
+- **Data mapper** - handling of received messages via connector and mapping them to a target data format for Cumulocity IoT.
   Also includes an expression runtime [JSONata](https://jsonata.org) to execute expressions
-* **C8Y client** - implements part of the Cumulocity IoT REST API to integrate data
-* **REST endpoints** - custom endpoints which are used by the MQTT Frontend or can be used to add mappings programmatically
-* **Mapper frontend** - A plugin for Cumulocity IoT to provide an UI for MQTT Configuration & Data Mapping
+- **C8Y client** - implements part of the Cumulocity IoT REST API to integrate data
+- **REST endpoints** - custom endpoints which are used by the MQTT Frontend or can be used to add mappings programmatically
+- **Mapper frontend** - A plugin for Cumulocity IoT to provide an UI for MQTT Configuration & Data Mapping
 
-> **Please Note:** When using MQTT or any other Message Broker beside MQTT Service you need to provide this broker available yourself to use the Dynamic Mapper.
+> **Please Note:** When using MQTT or any other Message Broker beside MQTT Service you need to provide this broker available yourself to use the Dynamic Data Mapper.
 
 The mapper processes messages in both directions:
+
 1. `INBOUND`: from Message Broker to C8Y
 2. `OUTBOUND`: from C8Y to Message Broker
 
-The Dynamic Mapper can be deployed as a **multi tenant microservice** which means you can deploy it once in your enterprise tenant and subscribe additional tenants using the same hardware resources.
+The Dynamic Data Mapper can be deployed as a **multi tenant microservice** which means you can deploy it once in your enterprise tenant and subscribe additional tenants using the same hardware resources.
 It is also implemented to support **multiple broker connections** at the same time. So you can combine multiple message brokers sharing the same mappings.
-This implies of course that all of them use the same topic structure and payload otherwise the mappings will fail.
+This implies of course that all of them use the same topic structure and payload otherwise the mappings will fail.

INSTALLATION.md

@@ -3,8 +3,9 @@
 ## Prerequisites
 
 To use the mapper you must have the following:
-* a Cumulocity Tenant with **microservice** feature subscribed (if not available please ask your contact or support)
-* Admin privileges for your user in the tenant
+
+- a Cumulocity Tenant with **microservice** feature subscribed (if not available please ask your contact or support)
+- Admin privileges for your user in the tenant
 
 ## Installation
 
@@ -17,6 +18,7 @@ Both are provided as binaries in [releases](https://github.com/SoftwareAG/cumulo
 Download the binaries from the latest release and upload them to your Cumulocity IoT Tenant.
 
 ## Permissions
+
 The solution defines one role:`ROLE_MAPPING_ADMIN` that must be assigned to the user accessing the Dynamic Mapping app.
 
 ### Microservice
@@ -26,21 +28,20 @@ In your Enterprise Tenant or Tenant navigate to "Administration" App, go to "Eco
 Select the `dynamic-mapping-service.zip`.
 Make sure that you subscribe the microservice to your tenant when prompted
 
-
 ### Web app
 
 The frontend can be used in two variants in your tenant:
+
 1. As a **UI Plugin** to extend existing applications
 2. As a **Blueprint** standalone Application selectable from the App switcher
 
 #### Community store (Preferred)
 
-
-
 The Web App is part of the community store and should be available directly in your tenant under
 "Administration" -> "Ecosystem" -> "Extensions" -> "Dynamic-mapping". Here you have the choice to install it as a plugin or as a blueprint app.
 
 ##### Plugin
+
 > **_NOTE:_** For a plugin we need to clone the Administration app to add the plugin to it
 
 Go to "All Applications" and click on "Add Application". Select "Duplicate existing application" and afterward "Administration".
@@ -50,21 +51,23 @@ Go to "All Applications" and click on "Add Application". Select "Duplicate exist
 </p>
 <br/>
 
-Now select the cloned Administration App and go to the "Plugin" Tab. Click on "Install Plugin" and select "Dynamic Mapper Widget"
+Now select the cloned Administration App and go to the "Plugin" Tab. Click on "Install Plugin" and select "Dynamic Data Mapper Widget"
 
 <p align="center">
 <img src="resources/image/Dynamic_Mapper_Installation_Plugin.png" style="width: 50%;" />
 </p>
 <br/>
 
 After successfully adding the plugin you need to refresh the Administration App by pressing F5 and you should see a new navigation entry "Dynamic Mapping"
+
 <p align="center">
  <img src="resources/image/Dynamic_Mapper_WebAppPlugin.png" style="width: 60%;" />
 </p>
 
 ##### Blueprint
 
 For the blueprint go to "Administration" -> "Ecosystem" -> "Dynamic-mapping" -> "Deploy application"
+
 <p align="center">
 <img src="resources/image/Dynamic_Mapper_BlueprintDeploy.png" style="width: 50%;" />
 </p>
@@ -83,4 +86,4 @@ If you made changes or your want to upload the plugin manually you can do that b
 1. In "Administration" App go to "Ecosystem" -> "Packages" and click on "Add Application" on the top right.
 2. Select `dynamic-mapping.zip` and wait until it is uploaded.
 
-Follow the steps from the point above to assign the plugin to your Administration App.
+Follow the steps from the point above to assign the plugin to your Administration App.

LIMITATIONS.md

@@ -3,17 +3,20 @@
 As we already have a very good C8Y API coverage for mapping not all complex cases might be supported. Currently, the
 mappings to the following C8Y APIs are supported:
 
-* inventory
-* events
-* measurements
-* alarms
-* operations (outbound to devices)
+- inventory
+- events
+- measurements
+- alarms
+- operations (outbound to devices)
 
 A mapping is defined of mapping properties and substitutions. The substitutions are mapping rules copying date from the incoming payload to the payload in the target system. These substitutions are defined using the standard JSONata as JSONata expressions. These JSONata expressions are evaluated in two different libraries:
+
 1. `dynamic-mapping`: (nodejs) [npmjs JSONata](https://www.npmjs.com/package/jsonata) and
 2. `dynamic-mapping-service` (java): [JSONata4Java](https://github.com/IBM/JSONata4Java)
    Please be aware that slight in differences in the evaluation of these expressions exist.
 
 Differences in more advanced expressions can occur. Please test your expressions before you use advanced elements.
 
-For MQTT Service currently no wildcards topics (e.g. `topic/#` or `topic/+` ) for Inbound Mappings / Subscriptions are allowed.
+For Cumulocity MQTT Service currently no wildcards topics (e.g. `topic/#` or `topic/+` ) for Inbound Mappings / Subscriptions are allowed.
+
+The [java library for JSONata](https://github.com/IBM/JSONata4Java) uses the words `and`, `or ` and `in` as reserved words in their [expression language](https://github.com/IBM/JSONata4Java/issues/317), hence they can be used as key in an JSON payload, see [issue](https://github.com/SoftwareAG/cumulocity-dynamic-mapper/issues/230).

README.md

@@ -2,26 +2,27 @@
 
 ## Overview
 
-The Cumulocity Dynamic Mapper addresses the need to get **any** data provided by a message broker mapped to the Cumulocity IoT Domain model in a zero-code approach.
+The Cumulocity Dynamic Data Mapper addresses the need to get **any** data provided by a message broker mapped to the Cumulocity IoT Domain model in a zero-code approach.
 It can connect to multiple message brokers likes **MQTT**, **MQTT Service** , **Kafka** and others, subscribes to specific topics and maps the data in a graphical editor to the domain model of Cumulocity.
 
 Per default the followings connectors are supported
-* **MQTT** - any MQTT Broker
-* **Cumulocity IoT MQTT Service** - Cumulocity IoT built-in MQTT Broker 
-* **Kafka** - Kafka Broker
 
-Using the Cumulocity Dynamic Mapper you are able to connect to almost any message broker and map any payload on any topic dynamically to
+- **MQTT** - any MQTT Broker
+- **Cumulocity IoT MQTT Service** - Cumulocity IoT built-in MQTT Broker
+- **Kafka** - Kafka Broker
+
+Using the Cumulocity Dynamic Data Mapper you are able to connect to almost any message broker and map any payload on any topic dynamically to
 the Cumulocity IoT Domain Model in a graphical editor.
 
 Here are the **core features** summarized:
 
-* **Connect** to multiple message broker of your choice at the same time.
-* **Map** any data to/from the Cumulocity IoT Domain Model in a graphical way.
-* **Bidirectional mappings** are supported - so you can forward data to Cumulocity or subscribe on Cumulocity data and forward it to the broker
-* **Transform** data with a comprehensive expression language supported by [JSONata](https://jsonata.org/) 
-* **Multiple payload formats** are supported, starting with **JSON**, **Protobuf**, **Binary**, **CSV**.
-* **Extend**  the mapper easily by using payload extensions or the provided connector interface
-* Full support of **multi-tenancy** - deploy it in your enterprise tenant and subscribe it to sub-tenants.
+- **Connect** to multiple message broker of your choice at the same time.
+- **Map** any data to/from the Cumulocity IoT Domain Model in a graphical way.
+- **Bidirectional mappings** are supported - so you can forward data to Cumulocity or subscribe on Cumulocity data and forward it to the broker
+- **Transform** data with a comprehensive expression language supported by [JSONata](https://jsonata.org/)
+- **Multiple payload formats** are supported, starting with **JSON**, **Protobuf**, **Binary**, **CSV**.
+- **Extend** the mapper easily by using payload extensions or the provided connector interface
+- Full support of **multi-tenancy** - deploy it in your enterprise tenant and subscribe it to sub-tenants.
 
 <br/>
 <p align="center">
@@ -31,54 +32,56 @@ Here are the **core features** summarized:
 
 ## Installation
 
-Please check the [Installation Guide](/INSTALLATION.md) to find out how you can install the Dynamic Mapper.
+Please check the [Installation Guide](/INSTALLATION.md) to find out how you can install the Dynamic Data Mapper.
 
 ## User Guide
 
-Please check the [User Guide](/USERGUIDE.md) to find a comprehensive guidance how to use the Dynamic Mapper.
+Please check the [User Guide](/USERGUIDE.md) to find a comprehensive guidance how to use the Dynamic Data Mapper.
 
 ## Architecture
 
 Please check the [Architecture overview](/ARCHITECTURE.md) if you are eager to understand how it is implemented.
 
 ## API
 
-Please check the [REST API](/API.md) provided by the Dynamic Mapper. 
+Please check the [REST API](/API.md) provided by the Dynamic Data Mapper.
 
 ## Extensions
 
-The Dynamic Mapper can be extended on multiple layers. Check out the [Extensions Guide](/EXTENSIONS.md) if you want to add customer mapper or connectors. 
+The Dynamic Data Mapper can be extended on multiple layers. Check out the [Extensions Guide](/EXTENSIONS.md) if you want to add customer mapper or connectors.
 
 ## Limitations
 
-Please check the current [Limitations](/LIMITATIONS.md) of the Dynamic Mapper.
+Please check the current [Limitations](/LIMITATIONS.md) of the Dynamic Data Mapper.
 
 ## Contribution
 
 We are always looking for additional [contribution](/CONTRIBUTING.md).
 
 ## Build & Deploy
 
-If you want to make changes to the code or configuration check out this [Build & Deploy guide](/BUILDDEPLOY.md) 
+If you want to make changes to the code or configuration check out this [Build & Deploy guide](/BUILDDEPLOY.md)
 
 ## Tests & Sample Data
 
 ### Load Test
-In the resource section you find a test profil [jmeter_test_01.jmx](./resources/script/performance/jmeter_test_01.jmx) using the performance tool ```jmeter``` and an extension for MQTT: [emqx/mqtt-jmeter](https://github.com/emqx/mqtt-jmeter).
+
+In the resource section you find a test profil [jmeter_test_01.jmx](./resources/script/performance/jmeter_test_01.jmx) using the performance tool `jmeter` and an extension for MQTT: [emqx/mqtt-jmeter](https://github.com/emqx/mqtt-jmeter).
 This was used to run simple loadtest.
 
 ## Setup Sample mappings
 
 A script to create sample mappings can be found [here](./resources/script/mapping/import_mappings_01.py).
 You have to start it as follows:
+
 ```
 #python3 resources/script/mapping/import_mappings_01.py -p <YOUR_PASSWORD> -U <YOUR_TENANT> -u <YOUR_USER> -f resources/script/mapping/sampleMapping/sampleMappings_02.json
 ```
 
 The mappings with inputs and substitutions are explained in the [sample document](./resources/script/mapping/sampleMapping/sampleMappings_02.html).
 
-______________________
+---
+
 These tools are provided as-is and without warranty or support. They do not constitute part of the Software AG product suite. Users are free to use, fork and modify them, subject to the license agreement. While Software AG welcomes contributions, we cannot guarantee to include every contribution in the master project.
 
 Contact us at [TECHcommunity](mailto:technologycommunity@softwareag.com?subject=Github/SoftwareAG) if you have any questions.
-

dynamic-mapping-service/src/main/java/dynamic/mapping/core/BootstrapService.java

@@ -203,7 +203,7 @@ public void initializeOutboundMapping(String tenant, ServiceConfiguration servic
 
     //shutdownAndRemoveConnector will unsubscribe the subscriber which drops all queues
     public void shutdownAndRemoveConnector(String tenant, String connectorIdent) throws ConnectorRegistryException {
-        connectorRegistry.unregisterClient(tenant, connectorIdent);
+        //connectorRegistry.unregisterClient(tenant, connectorIdent);
         ServiceConfiguration serviceConfiguration = serviceConfigurationComponent.getServiceConfiguration(tenant);
         if (serviceConfiguration.isOutboundMappingEnabled()) {
             configurationRegistry.getNotificationSubscriber().unsubscribeDeviceSubscriberByConnector(tenant, connectorIdent);