"Plot log file" page crashes after too many data are collected.

[sebi5361]

I am logging data on a SD card.
I am plotting those data using the "Plot log file" webpage (/DG option):

It works perfectly till a certain moment in time when data must be too numerous as after clicking on the "Plot log file" I reach this page:

... that remains the same for a long time till it crashes:

In my situation this occurs after 6 parameters being logged every 30 seconds for about 2 days.
And changing the time interval in that webpage before it crashes doesn't prevent it from crashing anyway (I'll test with a URL command that fixes the time interval to see if it works though).

• It would be nice to estimate a maximum time interval that can be plotted without crashing that webpage and to set it as a limit.
• Additionally, it would be nice to have the possibility to select a different sampling period for the data being sent by MQTT and the one being logged on the SD card.

[fredlcore]

We have discussed this at length here: #539 and here: #542
In all our testings the crashes you mention do not occur (at this rather "short" interval).
Several people have requested a different log interval, but I don't see the rationale behind that: SD card logging was introduced before we had MQTT, so that you could have some kind of graphical display at least. Once you log externally, there is simply no use-case for SD card logging anymore, since storage capacity, access speed and analytical features will always be (much) less effective compared to any home automation system you will have to be using in order to work with MQTT. If you want to have your own kind of MQTT log interval, this could always be achieved with custom code snippets, and maybe I'll write one to show how this would be done at some point, but for a more general solution that just adds clutter in the configuration for most users, I see no justification...

[sebi5361]

A look at the time interval in the picture above shows it's only two days.
48 h * 60 min * 2 samples/min * 6 simultaneous samples = 34560 samples.
For me it is not clear why you would spend time on interfacing an SD card adapter if you believe the SD card logging is an outdated feature. But reading the posts you mentioned I understand improving the code is somehow complicated.
Having that custom code snippet would be of great help.

[fredlcore]

If you mean our previous discussion regarding the SD card adapter for the NodeMCU, that doesn't really cost me a lot of time compared to refactoring the code the way you suggested it (plus the knowledge of adding SD card functionality to a NodeMCU device can come in handy for other projects). I didn't say that SD card logging is outdated, I said that there is no use-case once you do external logging via MQTT. There are, however, a number of cases where that is not easily possible (for example when running at at site with no (permanent) internet connection, such as a holiday home) and where SD card logging is a viable option. But in that case you don't need to differentiate between SD and MQTT logging intervals.

As for your crashes, it's hard to diagnose from afar because as shown in the mentioned threads, we have done testing with far greater datasets without any problems: #539 (comment)
At the least I would need is a SerMo log before, during and after the request for the graph display. The only thing I could act upon is if there is actually a crash due to a bug in the code. If BSB-LAN, however, keeps on running, then it's a network/browser issue on your side.

[sebi5361]

Yes I was referring to our previous discussion; I get your point.
My issue seems to be related to the browser type as with Chrome it crashes but with Firefox it works:

Thanks for BSB-LAN, a great piece of software!

[fredlcore]

You're welcome :) - as for the browser crash in Chrome, I remember that sometimes users have reported issues with different browsers (have to check if it was always Chrome), but I wonder if it's just a timeout problem or something else. Could you measure after which time Chrome crashes? If it's a "round" number of seconds/minutes, it might be a setting that can simply be adjusted. I'm not sure if BSB-LAN could change something in the code to prevent that because there is a continuous stream of data coming from BSB-LAN to the browser...
EDIT: @DE-cr did tests with Chromium here without any problems: #539 (comment) - did you also use the filtering that limits to 10k lines or did you download the full-size log?

[DE-cr]

If it works in Firefox, but crashes in Chrome, it's likely to be a browser issue, not a problem with the BSB-LAN code, and as you seem to have Firefox available, I'd just use that then.

What you could do to narrow the problem down is:

• see if /D (instead of /DG) works alright
• see if loading the /D output in the datalog viewer from the scripts directory crashes

How much RAM does Chrome have available on your system?

As for the separate log intervals for logging and MQTT: I do something similar by using custom code to send data via UDP to interested devices in my network, at a different interval from what I use for logging in BSB-LAN (different parameters, too) -> https://github.com/DE-cr/BSBmonCR, using "Variant 2". The logging in BSB-LAN I mostly use to analyze the boiler's behavior (and I re-configure the logging every once in a while), my custom one for "home automation" (well, mostly display) purposes. I very much appreciate that I can have the two separate.

P.s. Your two days log should be about 2.5 MB large. Mine cannot grow this big, as I have no SD card in my system. However, loading about 90,000 data samples (distributed over 7 parameters) in the external datalog viewer does not crash my current version of Chromium.

P.p.s. I'm not aware of a 10 KB filter in the current /DG implementation, only limiting it to a date range via the start/stop date controls.

[fredlcore]

It's not a 10kB, but 10k lines filter, as you mentioned in the linked comment. I have never checked upon that, just quoted the statement.

[DE-cr]

In the current implementation, that only exists in the external viewer, which upon loading a log file limits the start date to about 10,000 data samples before the end date, for a faster initial plot display.

There's no such thing left in BSB-LAN's /DG. For that there's only a days limit for the initial display: #define DEFAULT_DAYS_TO_PLOT "3" in BSB_LAN_config.h. Iirc we didn't expect that much data with most log settings and three days to initially be displayed per default. Lowering that setting to "1" could also help here. To then plot the previous day, one should first set the end date to that value, then the start date to the same value. This could be used as a workaround to the problem with Chrome on the thread starter's system.

[DE-cr]

Does the "Learn more" link offered by Chrome upon "crashing" provide helpful information?

[sebi5361]

Yes the "learn more" link offered by Chrome upon "crashing" provides helpful information!
https://support.google.com/chrome/answer/95669
... as I was able to successfully plot the log file in an Incognito window!
I might erase my Chrome history to see if the solution offered by Chrome works, but currently I am using firefox to plot the log file as I need to keep my Chrome history currently.

Yes the /D (instead of /DG) works alright and prints the log data directly in a Chrome webpage.

see if loading the /D output in the datalog viewer from the scripts directory crashes

I downloaded locally the log file using wget and then launched your BSBmonCR_log_viewer.html file on a local webserver in Chrome. Is it what you mean?
I got this output (same with Firefox):

[DE-cr]

The plot is not quite what I expected (obviously).
Could you post a shortened version of your D.csv file, e.g. the first 100 lines or so?

[sebi5361]

Here it is:
D100.csv

[DE-cr]

Hmm, your D100.csv loads okay with me, both in Firefox 123.0 and Chrome 120.0.6099.216:

[DE-cr]

Could you please try again with a current version of https://github.com/fredlcore/BSB-LAN/blob/master/BSB_LAN/scripts/BSB-LAN_datalog-viewer.html? Your version must be at least a year old, as there's no date controls to set the range to plot.

[sebi5361]

It works with the current version on my local server!!
(The version that doesn't work on my local server is the one in your GitHub repository)

[DE-cr]

Okay, that explains it: BSBmonCR is a completely different program (to run on a second ESP32, collecting data from a BSB-LAN unit via UDP) from BSB-LAN, and uses a completely different log format.

[sebi5361]

Oh OK.
This later issue might be related to my local server then, and is a different topic. I will try to investigate it however.

I found the reason why I couldn't plot the log data in Chrome!
As clearing my history wasn't enough, I deactivated extensions one by one to find that Click and Read CNRS was responsible for the issue:
Click and Read CNRS, 3.13.0, Extension that offers links to resources from documentary identifiers

So strange to me that this innocent extension is responsible for such an issue...
However, without it I can successfully plot my log data in Chrome!

[fredlcore]

Is there still something open for discussion here or can I close this topic?

[DE-cr]

Okay to close imo.

[sebi5361]

ok to close