Improvement/Provide a cron that upgrades the hub and the collections/parsers/scenarios/....

[blotus]

No description provided.

[cmiscloni]

+1

[okohll]

Does

#!/bin/bash
cscli hub update
cscli hub upgrade
cscli collections upgrade -a
cscli parsers upgrade --all
cscli scenarios upgrade -a
systemctl reload crowdsec

about cover everything?

[LaurenceJJones]

#1817 Fixed this

[mmetc]

There are still a few issues

• crowdsec does not reload the configuration after hub upgrade
• the package manager doesn't know about the cron job, so it won't be able to update it, or see if the user made changes
• wizard.sh is soon going be a /bin/sh script - not bash - for compatibility with other distributions. It won't be called nor sourced from the postinst file. Also passing arguments to sourced files (the -n for noop) only works with bash

For these reasons, it would be better to follow each distribution's guidelines and install the cron job like we would install any other file:

• https://fedoraproject.org/wiki/Packaging:CronFiles
• https://www.debian.org/doc/manuals/maint-guide/dother.en.html#crond

A minimal cron.daily/crowdsec-hub would look like:

#!/bin/sh
cscli --error hub update && cscli --error hub upgrade
systemctl reload crowdsec
exit 0

I'm not convinced that systems installed with wizard.sh should have cron jobs, but if they do, they might not have systemd. That part is WIP.

What do you think? @sabban ?

[mridullpandey]

hi @mmetc i want to take this up..

[mmetc]

Hi @mridullpandey

It's been merged here
https://github.com/crowdsecurity/crowdsec/blob/master/config/crowdsec.cron.daily

Any suggestion? We're releasing soon

Thanks

[tasiotas]

what about docker image?
could you include this cron in docker as well?

[mmetc]

@tasiotas there are several drawbacks to running a process manager and cron in a container, we'll try and find a better option for both docker and k8s

[jflattery]

@tasiotas there are several drawbacks to running a process manager and cron in a container, we'll try and find a better option for both docker and k8s

Is there a issue that can be tracked for this? Is there a "proper" way to do this as of today?

[zc-devs]

Combining #3256 and Kubernetes Cron Job I was able to import decisions via CLI. Maybe you can do it for Hub updates.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: cron
  namespace: crowdsec
spec:
  schedule: "1 1 * * *"
  jobTemplate:
    spec:
      template:
        metadata:
          name: cron
        spec:
          containers:
            - name: cron
              image: crowdsecurity/crowdsec:v1.6.3
              envFrom:
                - configMapRef:
                    name: cron-env
                - secretRef:
                    name: crowdsec-agent
              command:
                - "/bin/sh"
              args:
                - "/bin/cs-blocklists.sh"
              volumeMounts:
                - name: scripts
                  mountPath: /bin/cs-blocklists.sh
                  subPath: cs-blocklists.sh
          volumes:
            - name: scripts
              configMap:
                name: cron-file

apiVersion: v1
kind: ConfigMap
metadata:
  name: cron-env
  namespace: crowdsec
data:
  DISABLE_AGENT: "true"
  DISABLE_ONLINE_API: "true"
  DISABLE_LOCAL_API: "true"
  LOCAL_API_URL: http://lapi:8080

apiVersion: v1
kind: ConfigMap
metadata:
  name: cron-file
  namespace: crowdsec
data:
  cs-blocklists.sh: |
    ---
    #!/bin/sh

    /docker_start.sh

    blockLists='
    firehol-l1|https://iplists.firehol.org/files/firehol_level1.netset
    firehol-l2|https://iplists.firehol.org/files/firehol_level2.netset
    firehol-l3|https://iplists.firehol.org/files/firehol_level3.netset'

    echo "$blockLists" | while read -r blockList; do
      if [ -n "$blockList" ]; then
        listName=$(echo "$blockList" | cut -f 1 -d '|')
        echo "Processing $listName"
        listUrl=$(echo "$blockList" | cut -f 2 -d '|')
        listFileName="$listName.blocklist"
        wget "$listUrl" -O "$listFileName"

        echo "Importing single IPs from $listFileName"
        grep -E -i '^\d+\.\d+\.\d+\.\d+$' "$listFileName" > "$listFileName.ip"
        cscli decisions import --input "$listFileName.ip" --scope ip --duration 72h --reason "$listName" --format values
        rm -f "$listFileName.ip"

        echo "Importing ranges of IPs from $listFileName"
        grep -E -i '^\d+\.\d+\.\d+\.\d+/\d+$' "$listFileName" > "$listFileName.range"
        cscli decisions import --input "$listFileName.range" --scope range --duration 72h --reason "$listName" --format values
        rm -f "$listFileName.range"

        rm -f "$listFileName"
      fi
    done

    echo "Done"

apiVersion: v1
kind: Secret
metadata:
  name: crowdsec-agent
  namespace: crowdsec
stringData:
  AGENT_USERNAME: agent
  AGENT_PASSWORD: stub

[mmetc]

As of 1.6.3, if the file /etc/crowdsec/hub/.index.json is mounted and older than 24 hours, it runs a hub update / upgrade.

Which means a periodic container restart is the equivalent of the cron job. The crowdsec process would have to be restarted anyway, even when not using a container and even on k8s.