Cepheus-CEP multitenancy issue for multi-tenant provider

[pooja1pathak]

I have enabled the multi-tenant profile from application.properties:
spring.profiles.active=multi-tenant

and then uploaded config files having Fiware-Service and Fiware-ServicePath in Headers with serviceName, servicePath in providers section as:

A JSON file having name cep-testservice-testservicepath is created.

I have an entity of type Room having id Room1 in my Orion context broker.

These are my orion logs:

9/6/2018 5:12:57 PMtime=Thursday 06 Sep 11:42:57 2018.187Z | lvl=INFO | corr=ffbc3846-b1c9-11e8-843e-0298bbe0d856 | trans=1534488995-284-00000000503 | from=<cepheus-cep> | srv=pending | subsrv=/test | comp=Orion | op=rest.cpp[895]:servicePathSplit | msg=Service Path 0: '/test'
9/6/2018 5:12:57 PMtime=Thursday 06 Sep 11:42:57 2018.188Z | lvl=INFO | corr=ffbc3846-b1c9-11e8-843e-0298bbe0d856 | trans=1534488995-284-00000000503 | from=<cepheus-cep> | srv=testprovider | subsrv=/test | comp=Orion | op=connectionOperations.cpp[175]:collectionRangedQuery | msg=Database Operation Successful (query: { query: { $or: [ { _id.id: "Room1", _id.type: "Room" } ], _id.servicePath: { $in: [ /^/test$/ ] }, attrNames: { $in: [ "temperature", "floor" ] } }, orderby: { creDate: 1 } })
9/6/2018 5:12:57 PMtime=Thursday 06 Sep 11:42:57 2018.189Z | lvl=INFO | corr=ffbc3846-b1c9-11e8-843e-0298bbe0d856 | trans=1534488995-284-00000000503 | from=<cepheus-cep> | srv=testprovider | subsrv=/test | comp=Orion | op=connectionOperations.cpp[175]:collectionRangedQuery | msg=Database Operation Successful (query: { query: { $or: [ { _id.id: "Room1", _id.type: "Room" } ], _id.servicePath: { $in: [ /^/test$/ ] } }, orderby: { creDate: 1 } })
9/6/2018 5:12:57 PMtime=Thursday 06 Sep 11:42:57 2018.189Z | lvl=INFO | corr=N/A | trans=1534488995-284-00000000504 | from=pending | srv=pending | subsrv=pending | comp=Orion | op=logMsg.h[1832]:lmTransactionStart | msg=Starting transaction to http://<cepheus-cep>:8080/ngsi10/notifyContext
9/6/2018 5:12:57 PMtime=Thursday 06 Sep 11:42:57 2018.190Z | lvl=INFO | corr=N/A | trans=1534488995-284-00000000504 | from=pending | srv=pending | subsrv=pending | comp=Orion | op=httpRequestSend.cpp[599]:httpRequestSendWithCurl | msg=Sending message 153 to HTTP server: sending message of 594 bytes to HTTP server
9/6/2018 5:12:57 PMtime=Thursday 06 Sep 11:42:57 2018.191Z | lvl=INFO | corr=ffbc3846-b1c9-11e8-843e-0298bbe0d856 | trans=1534488995-284-00000000503 | from=<cepheus-cep> | srv=testprovider | subsrv=/test | comp=Orion | op=connectionOperations.cpp[373]:collectionInsert | msg=Database Operation Successful (insert: { _id: ObjectId('5b9112c1fab1f7327c478692'), expiration: 1536237777, reference: "http://<cepheus-cep>:8080/ngsi10/notifyContext", custom: false, throttling: -1, servicePath: "/test", status: "active", entities: [ { id: "Room1", isPattern: "false", type: "Room", isTypePattern: false } ], attrs: [ "temperature", "floor" ], metadata: [], blacklist: false, conditions: [], lastNotification: 1536234177, count: 1, expression: { q: "", mq: "", geometry: "", coords: "", georel: "" }, format: "JSON" })
9/6/2018 5:12:57 PMtime=Thursday 06 Sep 11:42:57 2018.191Z | lvl=INFO | corr=ffbc3846-b1c9-11e8-843e-0298bbe0d856 | trans=1534488995-284-00000000503 | from=<cepheus-cep> | srv=testprovider | subsrv=/test | comp=Orion | op=logMsg.h[1916]:lmTransactionEnd | msg=Transaction ended
9/6/2018 5:12:57 PMtime=Thursday 06 Sep 11:42:57 2018.364Z | lvl=INFO | corr=N/A | trans=1534488995-284-00000000504 | from=pending | srv=pending | subsrv=pending | comp=Orion | op=httpRequestSend.cpp[620]:httpRequestSendWithCurl | msg=Notification Successfully Sent to http://<cepheus-cep>:8080/ngsi10/notifyContext
9/6/2018 5:12:57 PMtime=Thursday 06 Sep 11:42:57 2018.364Z | lvl=INFO | corr=N/A | trans=1534488995-284-00000000504 | from=pending | srv=pending | subsrv=pending | comp=Orion | op=logMsg.h[1916]:lmTransactionEnd | msg=Transaction ended

And these are my Cepheus-CEP log:

2018-09-06 17:12:56.550  INFO 10680 --- [nio-8080-exec-2] c.e.e.c.service.EPServiceProviderImpl    : Initializing engine URI 'testservice/testservicepath' version 5.3.0
2018-09-06 17:12:56.651  INFO 10680 --- [nio-8080-exec-2] c.o.cepheus.cep.EsperEventProcessor      : Apply configuration
2018-09-06 17:12:56.652  INFO 10680 --- [nio-8080-exec-2] c.o.cepheus.cep.EsperEventProcessor      : Add new event type: EventType{id='Room1', type='Room', isPattern=false, attributes=[Attribute{name='temperature', type='double', metadata=[], jsonpath='null'}, Attribute{name='floor', type='string', metadata=[], jsonpath='null'}]}
2018-09-06 17:12:56.682  INFO 10680 --- [nio-8080-exec-2] c.o.cepheus.cep.EsperEventProcessor      : Add new event type: EventType{id='FloorX', type='Floor', isPattern=false, attributes=[Attribute{name='temperature', type='double', metadata=[], jsonpath='null'}]}
2018-09-06 17:12:56.683  INFO 10680 --- [nio-8080-exec-2] c.o.cepheus.cep.EsperEventProcessor      : Add new statement: INSERT INTO Floor SELECT floor as id, avg(temperature) as temperature FROM Room.win:time(10 min) GROUP BY floor OUTPUT LAST EVERY 10 sec
2018-09-06 17:12:57.080  INFO 10680 --- [nio-8080-exec-2] c.o.c.cep.persistence.JsonPersistence    : Save configuration in C:\Users\<user-name>\AppData\Local\Temp\/cep-testservice-testservicepath.json
2018-09-06 17:12:57.082  INFO 10680 --- [taskScheduler-1] c.o.cepheus.cep.SubscriptionManager      : Launch of the periodic subscription task at 2018-09-06T11:42:57.081Z
2018-09-06 17:12:57.229  INFO 10680 --- [nio-8080-exec-3] c.e.e.c.service.EPServiceProviderImpl    : Initializing engine URI 'testprovider/test' version 5.3.0
2018-09-06 17:12:57.333  INFO 10680 --- [nio-8080-exec-3] c.o.cepheus.cep.EsperEventProcessor      : EventIn: Event{type='Room', values={temperature=30.0, id=Room1, floor=aaa}}
2018-09-06 17:12:57.339 ERROR 10680 --- [nio-8080-exec-3] c.o.c.cep.controller.NgsiController      : Event processing error: {}

com.orange.cepheus.cep.exception.EventProcessingException: Event type named 'Room' has not been defined or is not a Map event type, the name 'Room' has not been defined as an event type
	at com.orange.cepheus.cep.EsperEventProcessor.processEvent(EsperEventProcessor.java:190)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
	at com.sun.proxy.$Proxy46.processEvent(Unknown Source)
	at com.orange.cepheus.cep.controller.NgsiController.notifyContext(NgsiController.java:62)
	at com.orange.ngsi.server.NgsiBaseController.notifyContextRequest(NgsiBaseController.java:54)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:776)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:705)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:644)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:291)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.orange.cepheus.cep.tenant.TenantFilter.doFilter(TenantFilter.java:82)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:102)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:85)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:90)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:516)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1086)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:659)
	at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1558)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1515)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Unknown Source)

Observations:

1. Subscription is done for the entity.
2. notifyContext is send to Cepheus-CEP from orion.
3. On having EventIn NgsiController throws an Event processing error saying Event type named 'Room' has not been defined or is not a Map event type, the name 'Room' has not been defined as an event type.

Although if i remove serviceName, servicePath fromproviders section in configuration, the same request works fine and does not give any error.

Should it be concluded that there is no function for handling multi-tenant provider in configuration of multi-tenant Cepheus-CEP?

Issues which I refered are: #60 and #69

[marcc-orange]

The configuration you have set (having providers with custom service/servicePath) is for a single tenant on the Cepheus side to access data from two tenants on the Orion side, so you must not enable enable multi-tenant mode for this use case.

In multi-tenant mode, Cepheus expects each tenant to have their own configurations.
You have to send two configurations, one for each tenant, using the Fiware-Service and Fiware-ServicePath headers in the POST request you make to configure the two tenants.

For tenant testProvider/test:

POST cepheus:8080/v1/admin/config
Fiware-Service: testProvider
Fiware-ServicePath: /test

{
  "host":"http://localhost:8080",
  "in":[...]
  "out":[...]
}

For tenant testBroker1/:

POST cepheus:8080/v1/admin/config
Fiware-Service: testBroker1
Fiware-ServicePath: /

{
  "host":"http://localhost:8080",
  "in":[...]
  "out":[...]
}

[pooja1pathak]

Hello @marcc-orange ,
Thanks for your answer.
I was trying to combine multi-tenancy feature of Cepheus CEP with multi-tenant provider Orion.

I have done the same as you suggested.

For tenant testservice/testservicepath:

POST cepheus:8080/v1/admin/config
Fiware-Service: testservice
Fiware-ServicePath: /testservicepath

{
  "host":"http://localhost:8080",
  "in":[
      ...
      "providers":[
      	{ 
          "url":"http://localhost:1026",
          "serviceName": "testprovider",
          "servicePath": "/test"
        }
      ],
      ...
   ]
  "out":[
      ...
      "brokers":[
        {
          "url":"http://localhost:1026",
          "serviceName": "testbroker1",
          "servicePath": "/"
        }
      ]
   ]
}

A JSON file having name cep-testservice-testservicepath is created as a result of the above request.

I observed an Event processing error saying Event type named 'Room' has not been defined or is not a Map event type, the name 'Room' has not been defined as an event type. As described above with logs.

The same usecase works fine if I remove serviceName and servicePath from the provider section of in.

I doubt if Cepheus-CEP does not process Event for multi-tenant provider when multi-tenant profile is enabled in application.properties?

[marcc-orange]

Sorry, but I do not understand what you are trying to do here.
The way multi-tenancy was defined in Cepheus is to have a 1-to-1 mapping between Orion and Cepheus tenants (if tenant A exists on Orion, then it shoud exist on Cepheus).

I don't think it is possible to make the multi-tenant feature of Cepheus and the service/servicePath feature of providers work at the same time.

Here is an example:

Suppose you have some entity "Room" on a tenant A (defined by service: A, ignoring servicePath here) on Orion.
In a multi-tenant enabled Cepheus, a notification for this "Room" entity should be sent to the corresponding tenant A inside Cepheus.

If a Cepheus tenant B attempts to subscribe to an entity "Room" with a provider to service A, how would Cepheus know if it must "route" the notification to tenant B and not to tenant A ?

If you setup a mechanism where Cepheus tracks its subscriptions across its tenants, how can you garantee no conflicts can occur ? A tenant A and B on Cepheus could then both subscribe to a "Room" from tenant A on Orion, who is updated when a notification occurs ? A or B, or both ?

[pooja1pathak]

I have verified 1-to-1 mapping between Orion and Cepheus tenants as per your comment. The functionality of multi-tenancy is working fine with same service/servicePath in Orion and Cepheus.

As you said The way multi-tenancy was defined in Cepheus is to have a 1-to-1 mapping between Orion and Cepheus tenants (if tenant A exists on Orion, then it shoud exist on Cepheus). , is there any document available for this. If yes then please provide the links.

If there is no documentation for multi-tenancy in Cepheus-CEP, I want to contribute for the same.

[marcc-orange]

@pooja1pathak Sadly, the multi-tenancy is still an undocumented feature. Mostly because the feature itself is incomplete, lacks testing, as well as most importantly, unit tests to prove it works as intended.
You are more than welcomed to add some documentation, but this will not be enough as the feature itself still needs more work before being considered in a "production ready" state.

The first part to documentation is adding an entry to https://github.com/Orange-OpenSource/fiware-cepheus/blob/master/doc/admin/cep.md to explain that there is a spring.profiles.active=multi-tenant setting to the application.properties files, explicitly stating that this feature is disabled by default and not yet "production ready".

The second part is adding a dedicated multi-tenand.md to https://github.com/Orange-OpenSource/fiware-cepheus/tree/master/doc/cep to explain how it works and how it relates to Orion tenants. It should explain the current limitations:

• the incompatibility between multi tenancy and serviceName/servicePath defined in provideds
• the outstanding bugs when using multi tenancy with Orion, mainly Incorrect processing for notification about changes in multiple context entities. #73.