Week 43 2024 routine

[kiwixbot]

Check nodes free space

df -h / && df -h /data

• create a report in issue comment

Nodes system upgrades

apt update && apt upgrade

• run systematically the upgrade on bastion, stats, services, storage, demo, mirrors-qa nodes
• check for and apply important security upgrade on worker nodes asap (imager-worker, ondemand, sisyphus)

(regular workers updates are done separately on a monthly basis for worker nodes to not impact production)

Backups

• Ensure all borg repositories are being updated

k8s cluster

• Check Pod errors or in CrashLoopBackoff

k get pods -A -o wide|grep -E 'Error|Crash'

• Check Pod restarts

k get pods -A -o wide | pyp -i 'print("\n".join([line for line in l if re.split(r"\s+", line)[4] != "0"]))'

• Check if k8s should/could be upgraded

curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/clusters/$KIWIX_PROD_CLUSTER | jq ".version,.upgrade_available"
curl -s -H "X-Auth-Token: $SCW_SECRET_KEY" https://api.scaleway.com/k8s/v1/regions/fr-par/versions | jq ".versions[].name"

• Upgrade k8s if applicable/possible

Stats

matomo - stats.kiwix.org

• Ensure download.kiwix.org stats are being recorded
• Check whether matomo should be upgraded

Grafana

• Alert list is normal
• Zimfarm dashboard is normal
• Jobs durations dashboard is normal
• There is no abnormal behaviors on cluster resources consumption
• All mdadm RAID arrays are OK (check all instances in the top left combobox)

Projects

• UptimeRobot has no alert
• zimit backlog is reasonable
• Cloud Code Signing Certificate usage is OK (look for Unused Signings under END ENTITY CERTIFICATES): we have 1,200 per year (August to August)
• Analyze zimit failed tasks
• PRs awaiting your review

Security

• Analyze/merge dependabot PRs

Note: this is an automatic reminder intended for the assignee(s).

[rgaudin]

Storage

Machine	Filesystem	Size	Used	Avail	Use%	Use change
bastion	/	37G	16G	20G	44%	-
stats	/	233G	115G	106G	53%	+4G
services	/	456G	330G	103G	76%	+1G
storage	/	147G	48G	92G	35%	+4G
storage	/data	30T	14T	15T	48%	-
imager-worker	/	1.9T	493G	1.3T	28%	don't care
sisyphus	/	233G	8.8G	212G	-	don't care
ondemand	/	25G	9.7G	14G	42%	-
ondemand	/data	216G	199M	205G	2%	don't care
mirrors-qa	/	38G	4.3G	32G	12%	-
demo	/	40G	9.7G	28G	27%	+0.4G
demo	/data	1.8T	1.2T	472G	72%	don't care

Misc

• library-generation was failing due to the Wiki Update failing (happens at the end of the process on purpose so no impact on library) because Kelson Wiki user we use was banned ('Kelson', 'Autoblocked because your IP address has been recently used by User:GPG7rb3li.). Amusingly enough, @kelson42 had blocked that user. This is the result of [wikis] mediawiki's not seeing users'IPs properly #39
• Opened SE watcher crashes on XML parsing errors openzim/zimfarm#1033
• Cloud Signing:
  -Signings This Month: 62
  • Unused Signings: 973

zimit

• Pipe has 6 tasks
• (1) x net::ERR_HTTP2_PROTOCOL_ERROR (Seed Page Load Failed)
• (3) x net::ERR_INVALID_AUTH_CREDENTIALS (Seed Page Load Failed)
• (1) x net::ERR_CONNECTION_REFUSED (Seed Page Load Failed)
• (2) x Navigation timeout of 90000 ms exceeded
• (1) x SyntaxError: Invalid regular expression (user defined Include)
• (8) x Browser disconnected (crashed?) webrecorder/browsertrix-crawler#706
• (2) x LookupError: unknown encoding: unicode openzim/warc2zim#331: u, u
• (1) x Rewriter choke on a relative URL inside a querystring openzim/warc2zim#380
• (1) x Direct fetch of page URL timed out (URL was set to a ZIP file)