Request for comments/help: Replacing data from log db

[mountaindude]

Background

As of May 2021, the centralised log db is no longer installed as part of Qlik Sense Enterprise on Windows (QSEoW).
See this post in the Qlik forums for more info and discussion.

Butler SOS has for a long time extracted data from log db and made it available in Grafana dashboards, in InfluxDB and as MQTT messages.
It has turned out to be very useful to have operational metrics (number of user sessions, CPU load etc) side-by-side with warnings and errors taken from log db.
It simply makes it much easier to understand what is causing the problems, poor performance or whatever might be causing issues.

How can Butler SOS deal with this?

The first good news is that most info - maybe even everything - found in log db is also available in the QSEoW log files.
If we can detect and forward the relevant data in the log files to Butler SOS we will have the same log info as what's available in log db.

The second piece of good news is that we can do this.
If we are to stay within the QSEoW ecosystem the best option is to create log4net log appenders that forward the log events we're interested in to Butler SOS. This has been thoroughly tested and works really well, with close to real time performance (usually <5 seconds until events show up in Butler SOS).
This means that log events will show up in Butler SOS way more quickly than with the previous solution (which relied on polling log db every few minutes). We will thus get closer to real-time access to log events as they happen.

Any challenges?

Actually, yes.

The log files contain LOTS of information. How can we know/decide which is relevant to bring into Butler SOS?
A reasonable starting point is to look at what's currently available in log db and make that data available in Butler SOS, with log files as the source.
Additional log events can be added later on.

You can help!

There's no public documentation on exactly what data goes into log db.
But maybe we can crowd source the important parts of this...

What log events should be available in Butler SOS?

Do you have thoughts on what log events or metrics you'd like to see in a tool like Butler SOS?
Feel free to share here in the forums!

🤓 Are you a techie? Share what's in your log db!

Do you have the tech skills to query the Postgres log db directly?

If so it would be awesome if you can run the query below and report the results back here or directly to goran -at- ptarmiganlabs -dot- com (if you feel the info should not got onto GitHub).
It's nothing sensitive really - just what errors and warnings exist in log db for the past 3 weeks, plus what parts of Sense created those warnings/errors. With this info it's then possible to bring as much of that info into Butler SOS from the log files.

The query looks like this:

select distinct
	process_name,
	logger,
	entry_level
  from public.log_entries
  where
	entry_level in ('WARN', 'ERROR') and
	(entry_timestamp > now() - INTERVAL '21 days' )
  order by
	process_name,
	logger,
	entry_level

The output can look like this:

[jurajmisina]

Hi Göran,
here's a snip from one of my environments. Is there any way you could grab platform metrics and platform events? Those are quite usefull as well and AFAIK not logged into file logs. Thanks.

[mountaindude]

Awesome @jurajmisina - thanks!

More specifically what do you mean by "platform metrics and events"?

[jurajmisina]

Hey. By "platform metrics and events" I mean messages from Windows Event log and Performance counters as described in Qlik Logging Service documentation under "Windows Event Log integration" section and "Windows Performance Monitor integration" section.

Sometimes it is quite useful to have these in one place with Qlik logs. I did have a situation when a Qlik server ran out of memory, the client blamed Qlik, but looking at PerfMonitor data (via Sense System Performance Analyzer) I could show that the Engine only consumed 20% of memory. Further investigation showed that the failure was caused by a SW unrelated to Qlik. You can achieve this by other means as well, but I got used to having it at one place, so I'm kind of looking for replacement.

[mountaindude]

I see. But isn't this pretty much what's reported by the health check API?

If so, it's already handled by Butler SOS.
That API is in fact the original reason why Butler SOS even exists - the project was started as a way to extract that info (memory used by engine, number of active users, what docs are loaded etc).
The reason why we needed that info was that it was early days for Sense, synced persistence was still all the rage and things just.... didn't quite work as expected. We needed a way to understand what was going on, and the health check API provided the needed data.

On the other hand... I think the healtcheck API only provide a subset of the data mentioned in the links you provided.
Your thoughts are very valid I'd say, I'll poke around to see if that data shows up in the log streams.

On a related note, it seems there are log messages flying around in QSEoW that never make it to disk log files...
By hooking into log4net and subscribing to log events you get a pretty complete (totally complete??) picture of all log messages.
And there seems to be messages that fly by that I can't find in the disk files.
Maybe not really that strange, still a bit of a surprise to me.

[mountaindude]

The above info has been used in recent (8.x) versions.
Specifically, the Log4Net log appender template files that ship with Butler SOS have been adopted to pick up relevant messages from the Sense logging framework and forward these to Butler SOS.