Auto deployment of EFK stack and Kibana dashboards with Kabanero Operator

[fwji]

Current behavior:

In OCP 4.2, cluster logging (EFK stack) is done by installing Elasticsearch Operator and Cluster Logging Operator through OLM. The user can then go to the Kibana route created during cluster logging installation, and import custom dashboards provided by other vendors.

The manual setup instruction for EFK stack is documented here on OCP doc site https://docs.openshift.com/container-platform/4.2/logging/cluster-logging-deploying.html

To Be:

We want the installation of EFK to be carried out by Kabanero Operator as an installation option. Users can also specify the dashboards to be preloaded by Kabanero installation.

Notes and other considerations:

There were a few major concerns with the existing OCP cluster logging: security, scalability and namespace separation. We've spoken with @ralanlittle and @jtmulvey recently on these issues. There doesn't seem to be much work on the logging side security based on Jim's feedback on the current state of cluster logging.

On the scalability side, the current cluster logging feature provided by OCP is for small to medium scale deployment and does not meet the scalability expected by Alan (cluster with 20000 apps). See the first statement in OCP document https://docs.openshift.com/container-platform/4.2/logging/cluster-logging-deploying-about.html
The primary limitation is the fact cluster logging operator is designed to have only one instance deployed to the default openshift-logging namespace. The user is unable to deploy a second EFK stack via cluster logging operator through OLM. We have spoken to Red Hat about supporting multiple EFK deployments, and we have to have a resolution in the future. We believe this feature can also resolve the namespace separation issue we have for large scale cluster.

While we wait for a multi-tenant solution for EFK logging, we are hoping the automatical deployment of EFK and dashboards can be implemented by Kabanero Operator.

[kaczyns]

Thanks for this requirement Frank. We understand the desire to make the EFK stack available to users without manual configuration. There are still some concerns here, particularly around having a separate EFK stack specifically for application use. We will continue to work thru the issues. Lets keep this Github issue updated as we work thru the issues.