update sync method and to python3.12

Author: akmalovaa

.env.example

@@ -0,0 +1,6 @@
+MIKROTIK_HOST=192.168.88.1
+MIKROTIK_USER=username
+MIKROTIK_PASSWORD=password
+BLOCKLIST_URL=http://YOUR_BLOCK_LIST/security/blocklist?ipv4only
+SYNC_INTERVAL_MIN=15
+LOG_LEVEL=INFO

Dockerfile

@@ -1,11 +1,11 @@
-FROM python:3.11.9-slim
+FROM python:3.12.5-slim
 
 WORKDIR /srv/
 
 COPY pyproject.toml .
 COPY poetry.lock .
 
-RUN apt update && apt upgrade -y && pip install poetry
+RUN apt update && pip install poetry
 
 RUN poetry config virtualenvs.create false
 RUN poetry install
@@ -14,4 +14,4 @@ RUN apt-get remove -y gcc cmake make libc-dev-bin libc6-dev
 RUN rm -rf /var/lib/apt/lists/* && apt-get autoremove -y && apt-get clean
 RUN pip uninstall pipenv poetry -y
 
-COPY . .
+COPY . .

README.md

@@ -1,33 +1,22 @@
-# MikroSecList
+# MikroSecList - Mikrotik Security List
 
-## Mikrotik + CrowdSec block lists sync
+[![crowdsec blocklist](./crowdsec_blocklist/crowdsec_blocklist.png)](https://github.com/akmalovaa/mikroseclist)
 
-This app constantly synchronizes and updates the firewall address list in mikrotik.
+CrowdSec Blocklist mirror synchronization to RouterOS firewall address list
 
-Actual dangerous IP addresses will already be in the blocked list
+- [DockerHub crowdsec](https://hub.docker.com/r/crowdsecurity/crowdsec)
+- [DockerHub blocklist-mirror](https://hub.docker.com/r/crowdsecurity/blocklist-mirror)
+- [Docs Blocklist mirror](https://docs.crowdsec.net/u/bouncers/blocklist-mirror#installation/)
 
-[CrowdSec Blocklist mirror](https://docs.crowdsec.net/u/bouncers/blocklist-mirror/#installation/)
+### Guide 
 
-Allows you to use a list of IP addresses to add
+Configure Blocklist Mirror or use your own list of IP addresses *(need HTTP format output list)*
 
 *`config.yml`*
 ```yaml | code
 blocklists:
-  format: mikrotik
+  format: plain_text
 ```
-Output lines for mikrotik, format is `/ip|/ipv6 firewall address-list add list={list_name} address={ip} comment="{scenario} for {duration}"`
-
-The list of dangerous IP addresses is very large ~ 25,000, when updated in this way, all addresses are deleted and added again. It's pointless to do this every time you update.
-
-This service only allows you to edit changes. Delete something, add something
-
-## Guide 
-
-### Prepare block list
-
-[docker-compose crowdsec-blocklist](https://github.com/akmalovaa/crowdsec-blocklist)
-
-Or use your own list of IP addresses, the list should be accessible by url
 
 ### Prepare Router OS
 
@@ -39,22 +28,29 @@ add name=Server common-name=server
 add name=Client common-name=client
 ```
 
-Certificates should be signed.
+Certificates should be signed. 
+**Change your RouterOS host address**
 ```
 /certificate
 sign CA-Template
 sign Client     
 sign Server ca-crl-host=192.168.88.1 name=ServerCA
 ```
 
-Enable API-SSL
+Enable API-SSL. **Change api access address**
 ```
 /ip service
 set api-ssl address=192.168.88.0/24 certificate=ServerCA
 ```
 
 ### Docker compose
 
+change `.env` file variables
+```
+cp .env.exmaple .env
+nano .env
+```
+
 build
 ```bash
 docker build . -t mikroseclist:latest
@@ -89,10 +85,40 @@ change environment variables and run:
 docker-compose up -d
 ```
 
-Change Mikrotik Firewall Rules
+After first syncing сhange Mikrotik Firewall Rules
 ```sh
 /ip firewall filter
 add action=accept chain=input src-address-list=access # access list optional
 add action=drop chain=input in-interface=ether1 src-address-list=block
 add action=drop chain=forward in-interface=ether1 src-address-list=block
-```
+```
+
+### Settings
+
+https://github.com/akmalovaa/mikroseclist/blob/main/mikroseclist/settings.py
+
+You can override this variables in the .env file
+
+
+## CrowdSec block lists sync
+
+You can use default **CrowdSec Blocklist mirror** format without `mikroseclist` service:
+
+This app constantly synchronizes and updates the firewall address list in mikrotik.
+
+Actual dangerous IP addresses will already be in the blocked list
+
+[CrowdSec Blocklist mirror](https://docs.crowdsec.net/u/bouncers/blocklist-mirror/#installation/)
+
+Allows you to use a list of IP addresses to add
+
+*`config.yml`*
+```yaml | code
+blocklists:
+  format: mikrotik
+```
+Output lines for mikrotik, format is `/ip|/ipv6 firewall address-list add list={list_name} address={ip} comment="{scenario} for {duration}"`
+
+The list of dangerous IP addresses is very large ~ 25,000, when updated in this way, all addresses are deleted and added again. It's pointless to do this every time you update.
+
+This service only allows you to edit changes. Delete something, add something

Taskfile.yml

@@ -1,8 +1,18 @@
-version: "3"
+version: '3'
+
+env:
+  IMAGE_TAG: 'mikroseclist:latest'
 
 tasks:
+  cli:
+    aliases: [bash, shell, exec]
+    desc: run temporary container and passthrough current directory
+    cmds:
+      - docker run --rm -it --env-file .env -v .:/srv $IMAGE_TAG bash
+
   build:
-    desc: "build Dockerfile"
-    dir: "{{.USER_WORKING_DIR}}"
     cmds:
-      - docker build . -t mikroseclist:latest
+      - docker build . -t $IMAGE_TAG
+
+# export PYTHONPATH='/srv/mikroseclist/'
+# python -m mikroseclist.main

compose-dev.yaml

@@ -1,5 +1,3 @@
-version: "3.9"
-
 services:
   mikroseclist:
     image: mikroseclist:latest
@@ -8,12 +6,14 @@ services:
       dockerfile: Dockerfile 
     container_name: mikroseclist
     command: ["python", "-m", "mikroseclist.main"]
+    # command: ["tail", "-f", "/dev/null"]
     environment:
+      LOG_LEVEL: DEBUG
       MIKROTIK_HOST: ${MIKROTIK_HOST:-'192.168.88.1'}
       MIKROTIK_USER: ${MIKROTIK_USER:-'admin'}
       MIKROTIK_PASSWORD: ${MIKROTIK_PASSWORD:-'password'}
       BLOCKLIST_URL: ${BLOCKLIST_URL}
-      SYNC_INTERVAL_MIN: 15
+      SYNC_INTERVAL_MIN: 2
     volumes:
       - ./:/srv/
     restart: on-failure

compose.yaml

@@ -1,14 +1,12 @@
-version: "3.9"
-
 services:
   mikroseclist:
     image: ghcr.io/akmalovaa/mikroseclist:latest
     container_name: mikroseclist
-    command: ["python", "-m", "mikroseclist.main"]
+    command: ["python", "-m", "mikroseclist"]
     environment:
       MIKROTIK_HOST: ${MIKROTIK_HOST:-'192.168.88.1'}
       MIKROTIK_USER: ${MIKROTIK_USER:-'admin'}
       MIKROTIK_PASSWORD: ${MIKROTIK_PASSWORD:-'password'}
-      BLOCKLIST_URL: 'http://blocklist:41412/security/blocklist?ipv4only'
+      BLOCKLIST_URL: ${BLOCKLIST_URL:-'http://blocklist:41412/security/blocklist?ipv4only'}
       SYNC_INTERVAL_MIN: 15
     restart: on-failure

crowdsec/crowdsec-blocklist-mirror.yaml crowdsec_blocklist/blocklist_config.yaml

@@ -1,5 +1,3 @@
-# Install guide https://github.com/akmalovaa/crowdsec-blocklist
-
 services:
   crowdsec:
     image: ${CROWDSEC_IMAGE_TAG:-crowdsecurity/crowdsec:latest}
@@ -36,7 +34,7 @@ services:
     ports:
       - 41412:41412
     volumes:
-      - ./crowdsec-blocklist-mirror.yaml:/etc/crowdsec/bouncers/crowdsec-blocklist-mirror.yaml
+      - ./blocklist_config.yaml:/etc/crowdsec/bouncers/crowdsec-blocklist-mirror.yaml
     healthcheck:
       test: wget -qO- http://localhost:41412/security/blocklist?ipv4only || exit 1
       start_period: 20s
@@ -45,7 +43,7 @@ services:
       retries: 5
     networks:
       crowdsec_net:
-        ipv4_address: 172.21.0.2
+        ipv4_address: 172.21.0.12
 
   mikroseclist:
     image: ${MIKROSECLIST_IMAGE_TAG:-ghcr.io/akmalovaa/mikroseclist:latest}
@@ -66,7 +64,7 @@ services:
     restart: unless-stopped
     networks:
       crowdsec_net:
-        ipv4_address: 172.21.0.3
+        ipv4_address: 172.21.0.13
 
 networks:
   crowdsec_net:

compose-crowdsec.yaml crowdsec_blocklist/compose.yaml

@@ -0,0 +1,101 @@
+import ipaddress
+import sys
+import time
+
+import requests
+from loguru import logger
+from schedule import every, repeat, run_pending
+
+from mikroseclist.mikrotik_client import MikroTikClient
+from mikroseclist.settings import settings
+
+logger.remove()
+logger.add(
+    sys.stderr,
+    level=settings.log_level,
+    format="{time:DD.MM.YY HH:mm:ss} {level} {message}",
+)
+
+mikrotik = MikroTikClient(
+    settings.mikrotik_host, settings.mikrotik_user, settings.mikrotik_password
+)
+
+
+def download_blocklist_file() -> None:
+    logger.info(f"Fetch blocklist from url: {settings.blocklist_url}")
+    try:
+        response = requests.get(settings.blocklist_url)
+    except requests.exceptions.RequestException as e:
+        logger.error(
+            f"Fetch blocklist from url: {settings.blocklist_url} exceptions: {e}"
+        )
+        sys.exit()
+    if response.status_code == 200:
+        with open(settings.blocklist_filename, "wb") as file:
+            file.write(response.content)
+        logger.debug(f"Save crowdsec blocklist to file: {settings.blocklist_filename}")
+    else:
+        logger.error("Addresses list could not be downloaded")
+    return
+
+
+def fetch_crowdsec_block_list() -> list:
+    logger.debug(f"Fetch data from file: {settings.blocklist_filename}")
+    block_list: list = []
+    download_blocklist_file()
+    with open(settings.blocklist_filename, "r") as file:
+        for line in file:
+            stripped_line = line.strip()
+            if stripped_line:
+                try:
+                    ipaddress.ip_address(stripped_line)
+                    block_list.append(stripped_line)
+                except ValueError:
+                    logger.warning(
+                        f"Invalid IP address found and skipped: {stripped_line}"
+                    )
+    return block_list
+
+
+def sync_addres_list() -> None:
+    logger.info(f"Run synchronization")
+    mikrotik_block_list: list = mikrotik.fetch_address_list()
+    crowdsec_block_list: list = fetch_crowdsec_block_list()
+    mikrotik_add_list: list = list(set(crowdsec_block_list) - set(mikrotik_block_list))
+    mikrotik_delete_list: list = list(
+        set(mikrotik_block_list) - set(crowdsec_block_list)
+    )
+    if mikrotik_add_list:
+        logger.debug(f"Found {len(mikrotik_add_list)} new addresses to add")
+        mikrotik.add_address_list(mikrotik_add_list)
+    else:
+        logger.debug(f"No addresses found to add")
+    if mikrotik_delete_list:
+        logger.debug(f"Found {len(mikrotik_delete_list)} new addresses to delete")
+        mikrotik.delete_address_list(mikrotik_delete_list)
+    else:
+        logger.debug(f"No addresses found to delete")
+    logger.info(
+        f"Successful synchronization. Total block list addresses: {len(crowdsec_block_list)}"
+    )
+
+
+@repeat(every(settings.sync_interval_min).minutes)
+def run_sync_list() -> None:
+    logger.info("sync mikrotik firewall address list")
+    sync_addres_list()
+
+
+def run_scheduler() -> None:
+    logger.info(
+        f"Run scheduler: sync address list every {settings.sync_interval_min} minutes"
+    )
+    while True:
+        run_pending()
+        time.sleep(60)
+
+
+if __name__ == "__main__":
+    logger.info("Mikrotik firewall address list synchronization started")
+    sync_addres_list()
+    run_scheduler()