Set up web analytics with GoAccess

[acka47]

Current status is at http://gaia.hbz-nrw.de/stats/

[acka47]

To Dos:

• Filter out all the URLs that are sideloaded when opening a web page, e.g. for NWBib:
  
• Provide yearly overview in addition to monthly view.

[Phu2]

The monthly access logs (eg. access_log-20230901) were only partially (01/Aug/2023 — 18/Aug/2023) included in the generated reports due to our preprocessing.
Example: for some reason grep ' www.lobid.org ' /tmp/access_log-20230901 didn't parsed the whole log file and output "grep: /tmp/access_log-20230901: Übereinstimmungen in Binärdatei". Solution: use grep --text or fgrep --text as we are searching for a fixed string.

[dr0i]

The binary data in the logs results from the server crash. --text is good!

[Phu2]

We could do bzfgrep --text on the compressed logs directly. It takes 7m3,410s compared to 2m51,289s for access_log-20230901(.bz2). What do you think?

[dr0i]

bzfgrep --text 👍

[Phu2]

I'm stuck at filtering out static files like *.png or *.css, eg.

grep --text -E -v "(robots.txt|.ico|.woff2|.ttf|.webp|.gif|.svg|.jpg|.png|.js|.css)" access_log_lobid-blog-20230901

works fine, but these files are still listed in the report generated by goaccess, see screenshot:

and i don't know why. @dr0i Can you help?

[dr0i]

I couldn't find a flaw in your code . Checking http://gaia.hbz-nrw.de/stats/lobid/access_log_lobid-blog-2023-09-01.html I cannot see e.g. any png. Did you check the proper output?

[Phu2]

Due to some obscure replacement of quotes in bash grep commands like the one above don't work as expected. As a workaround i'm calling grep directly in bash (not via variable nor array). Thx, @dr0i .

[Phu2]

All monthly and yearly reports for 2023 are beeing generated anew. It will take approx. 10 hours.

[Phu2]

Log files from 20230101 should be excluded from the yearly overview for 2023. Include files from 20240101 instead as they contain entries from december 2023.

[Phu2]

Yearly overviews are beeing generated anew.

[Phu2]

@acka47 Please review again.

[acka47]

This looks good to me now. Thanks! Can we make the stats available openly on the web so that NWBib editors can view it? I guess, there shouldn't be any problems re. privacy.

Furthermore, we will also need this for RPB (https://rpb.lobid.org/), RPPD (https://rppd.lobid.org/) and BiblioVino (https://wein.lobid.org/). LBZ partners just asked, see RPB-42. Should I open a separate issue for this or do we add this in the context of this issue?

[acka47]

I just talked about this issue with @Phu2 . And here are the next steps:

• Set up GoAccess for RPB (https://rpb.lobid.org/), RPPD (https://rppd.lobid.org/) and BiblioVino (https://wein.lobid.org/)
• Add stats for 2023-10 to 2023-12 and for 2023 as a whole
• Make all GoAccess pages publicly available
• Set up reverse proxies:
  • https://lobid.org/stats -> http://gaia.hbz-nrw.de/stats/lobid/
  • https://nwbib.de/stats -> http://gaia.hbz-nrw.de/stats/nwbib/
  • https://rpb.lobid.org/stats
  • https://rppd.lobid.org/stats
  • https://wein.lobid.org/stats

[Phu2]

Things to consider:

• Monthly reports are retrospective, eg. for january stats see access_log_nwbib-de-2023-02-01.html
• From 2023-11 onwards reports are based on dedicated access logs for each service. Before that we had one single large access log from which we filtered out the requests for each service.