elasticsearch compatibility above 7.0.0

[priamai]

I have noticed that the elast-server is pinned to pip packaege of elasticsearch to 7.0.0.
When I try to run with ES of latest version like 7.4.0 the server fails because doc_type is not supported anymore.
What's the best way to add an argument to match the ES version?
Open to suggestions.

[priamai]

Okay I can see there is a TODO comment here:
https://github.com/Yelp/elastalert/blob/ea62cf4bd1a62be3616f40992d088e58e2727f40/elastalert/create_index.py
is it tracked anywhere ?

[nsano-rururu]

doc_type is deprecated and will be removed in ES 8

[nsano-rururu]

I'm using elasticsearch 7.9.3 with pip packaege of elasticsearch to 7.0.0, but it works fine.

[nsano-rururu]

How can I get it stuck?
Do you do something that I can't reproduce?

[nsano-rururu]

Environment is docker

praecoapp/elastalert-server
praecoapp/praeco
elasticsearch 7.9.3
kibana 7.9.3

[nsano-rururu]

Currently, ElastAlert seems to be out of maintenance, so there are no signs of new features or bug fixes.

[priamai]

Yes I noticed this is what I tried to do but fails:

docker run -d --rm -p 9200:9200 -p 9300:9300\
  --network="host" --name="elasticsearch" -e "discovery.type=single-node"\
   docker.elastic.co/elasticsearch/elasticsearch:7.4.0


docker run -d --rm -p 3030:3030 -p 3333:3333 --network="host" \
    -v `pwd`/config/elastalert.yaml:/opt/elastalert/config.yaml \
    -v `pwd`/config/config.json:/opt/elastalert-server/config/config.json \
    -v `pwd`/rules:/opt/elastalert/rules \
    -v `pwd`/rule_templates:/opt/elastalert/rule_templates \
    --name elastalert bitsensor/elastalert:3.0.0-beta.0


docker run --rm -p 8080:8080 --network="host" \
    -v `pwd`/public/praeco.config.json:/var/www/html/praeco.config.json \
    -v `pwd`/nginx_config/nginx.conf:/etc/nginx/nginx.conf \
    -v `pwd`/nginx_config/default.conf:/etc/nginx/conf.d/default.conf \
    --name praeco praecoapp/praeco

Error:

WARNING: Published ports are discarded when using host network mode
Starting Nginx
nginx: [emerg] host not found in upstream "elastalert" in /etc/nginx/conf.d/default.conf:11

[nsano-rururu]

bitsensor / elastalert and servercentral / elastalert are not maintained.
The developer of servercentral / elastalert forks and maintains praecoapp / elastalert-server.
praecoapp / elastalert-server and praecoapp / praeco is also a co-maintainer for me.

Docker image name	tag	ElastAlert	Elasticsearch 7 Support	Remarks
bitsensor/elastalert	2.0.1	0.1.39	×	python 2.7
bitsensor/elastalert	lastet	0.1.39	×	python 2.7
bitsensor/elastalert	3.0.0-beta.0	0.2.0b2	〇	python 2.7
bitsensor/elastalert	3.0.0-beta.1	0.2.0b2	〇	python 2.7
servercentral/elastalert	latest	0.2.1	〇	python 3.6 bitsensor/elastalert fork Customize bug fix
daichi703n/elastalert	0.2.1-dev2	0.2.1	〇	python 3.6 servercentral/elastalert fork Customize bugfix
praecoapp/elastalert-server	latest	0.2.4	〇	python 3.8 servercentral/elastalert fork bug fix Library Update

[nsano-rururu]

default.conf, nginx.conf, praeco.config.json, api.config.json, elastalert.yaml are from praeco's github

/home/sano/dkwork2/es
|--Dockerfiles
|  |--Dockerfile.elastalert
|--docker-compose.yml
|--es
|  |--config
|  |  |--elasticsearch.yml
|  |--data
|--kibana
|  |--config
|  |  |--kibana.yml
|--praeco
|  |--bin
|  |  |--elastalert-start.sh
|  |  |--elastic_search_status.sh
|  |--config
|  |  |--api.config.json
|  |  |--elastalert.yaml
|  |--nginx_config
|  |  |--default.conf
|  |  |--nginx.conf
|  |--public
|  |  |--praeco.config.json
|  |--rule_templates
|  |--rules
|  |  |--BaseRule.config

Dockerfiles/Dockerfile.elastalert

FROM praecoapp/elastalert-server:latest

USER root

RUN apk update && \
    apk add vim bash curl && \
    rm -rf /var/cache/apk/*

ADD praeco/bin/elastalert-start.sh /usr/local/bin/
ADD praeco/bin/elastic_search_status.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/elastalert-start.sh 
RUN chmod +x /usr/local/bin/elastic_search_status.sh

USER node

ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]

es/config/elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1

kibana/config/kibana.yml

server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true

praeco/bin/elastalert-start.sh

#!/bin/bash

set -e

echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."

elastic_search_status.sh

echo "Starting ElastAlert!"
npm start

praeco/bin/elastic_search_status.sh

#!/bin/bash

set -e

if [ $# -gt 0 ]; then
  ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
  ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
  ES_URL="http://$ES_HOST:$ES_PORT"
else
  ES_URL="http://elasticsearch:9200"
fi

until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
  # printf '+' >&2
  sleep 1
done

echo "Elasticsearch is up and healthy at "$ES_URL"" >&2

docker-compose.yml

version: "3.7"
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.3
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - ES_JAVA_OPTS=-Xms256m -Xmx512m
      - discovery.type=single-node
    restart: always
    volumes:
      - ./es/data:/usr/share/elasticsearch/data
      - ./es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 180s

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.9.3
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  elastalert:
    container_name: elastalert
    build:
      context: .
      dockerfile: Dockerfiles/Dockerfile.elastalert
    image: elastalert-server:3.0.0
    ports:
      - 3030:3030
      - 3333:3333
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./praeco/config/elastalert.yaml:/opt/elastalert/config.yaml
      - ./praeco/config/api.config.json:/opt/elastalert-server/config/config.json
      - ./praeco/rules:/opt/elastalert/rules
      - ./praeco/rule_templates:/opt/elastalert/rule_templates
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:3030 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  praeco:
    container_name: praeco
    image: praecoapp/praeco:latest
    ports:
      - 8080:8080
    depends_on:
      - elastalert
    restart: always
    volumes:
      - ./praeco/public/praeco.config.json:/var/www/html/praeco.config.json
      - ./praeco/nginx_config/nginx.conf:/etc/nginx/nginx.conf
      - ./praeco/nginx_config/default.conf:/etc/nginx/conf.d/default.conf
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:8080 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

chmod -R 777 es/data
chmod -R 777 rules rule_templates
docker-compose up -d

docker logs -f elastalert

Giving Elasticsearch at  time to start...
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
Elasticsearch is up and healthy at http://elasticsearch:9200
Starting ElastAlert!

> @bitsensor/elastalert@0.0.14 start /opt/elastalert-server
> sh ./scripts/start.sh

18:05:17.808Z  INFO elastalert-server: Config:  No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
18:05:17.810Z  INFO elastalert-server: Config:  Proceeding to look for normal config file.
18:05:17.839Z  INFO elastalert-server: Config:  A config file was found in /opt/elastalert-server/config/config.json. Using that config.
18:05:17.904Z  INFO elastalert-server: Router:  Listening for GET request on /.
18:05:17.905Z  INFO elastalert-server: Router:  Listening for GET request on /status.
18:05:17.906Z  INFO elastalert-server: Router:  Listening for GET request on /status/errors.
18:05:17.907Z  INFO elastalert-server: Router:  Listening for GET request on /rules.
18:05:17.910Z  INFO elastalert-server: Router:  Listening for GET request on /rules/:id*.
18:05:17.911Z  INFO elastalert-server: Router:  Listening for POST request on /rules/:id*.
18:05:17.940Z  INFO elastalert-server: Router:  Listening for DELETE request on /rules/:id*.
18:05:17.941Z  INFO elastalert-server: Router:  Listening for GET request on /templates.
18:05:17.942Z  INFO elastalert-server: Router:  Listening for GET request on /templates/:id*.
18:05:17.943Z  INFO elastalert-server: Router:  Listening for POST request on /templates/:id*.
18:05:17.943Z  INFO elastalert-server: Router:  Listening for DELETE request on /templates/:id*.
18:05:17.945Z  INFO elastalert-server: Router:  Listening for PUT request on /folders/:type/:path*.
18:05:17.945Z  INFO elastalert-server: Router:  Listening for DELETE request on /folders/:type/:path*.
18:05:17.946Z  INFO elastalert-server: Router:  Listening for POST request on /test.
18:05:17.949Z  INFO elastalert-server: Router:  Listening for POST request on /silence/:path*.
18:05:17.956Z  INFO elastalert-server: Router:  Listening for GET request on /config.
18:05:17.957Z  INFO elastalert-server: Router:  Listening for POST request on /config.
18:05:17.958Z  INFO elastalert-server: Router:  Listening for POST request on /download.
18:05:17.959Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert.
18:05:17.960Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_status.
18:05:17.961Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/silence.
18:05:17.962Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_error.
18:05:17.962Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/past_elastalert.
18:05:17.967Z  INFO elastalert-server: Router:  Listening for GET request on /indices.
18:05:17.968Z  INFO elastalert-server: Router:  Listening for GET request on /mapping/:index.
18:05:17.969Z  INFO elastalert-server: Router:  Listening for POST request on /search/:index.
18:05:17.969Z  INFO elastalert-server: Router:  Listening for GET request on /config.
18:05:17.977Z  INFO elastalert-server: ProcessController:  Starting ElastAlert
18:05:17.978Z  INFO elastalert-server: ProcessController:  Creating index
18:05:19.206Z  INFO elastalert-server:
    ProcessController:  Elastic Version: 7.9.3
    Reading Elastic 6 index mappings:
    Reading index mapping 'es_mappings/6/silence.json'
    Reading index mapping 'es_mappings/6/elastalert_status.json'
    Reading index mapping 'es_mappings/6/elastalert.json'
    Reading index mapping 'es_mappings/6/past_elastalert.json'
    Reading index mapping 'es_mappings/6/elastalert_error.json'
    Index praeco_elastalert_status already exists. Skipping index creation.

[priamai]

Thanks ver much for your config!
I do notice some FATAL errors if you leave it running long enough on Kibana:

kibana           | 
kibana           |  FATAL  Error: Unknown configuration key(s): "cluster.name", "network.host", "discovery.zen.minimum_master_nodes". Check for spelling errors and ensure that expected plugins are installed.

Have you noticed those?

[nsano-rururu]

I woke up now due to lack of sleep.
Wrong copy of kibana.yml settings. I modified it to the actual setting.

[priamai]

@nsano-rururu thank you for all the support is working now I also woke up early!

I have some quick questions:

• if this is not mantained anymore servercentral / elastalert, does it mean that we will not have any future feature releases but just maybe ES compatibility releases?
• for the table praecoapp/elast-server the tickbox for ES7 is not enabled does it mean has not been tested extensively? Do you have some unit tests that you run to make sure the generated ruels are working? I could help with that since I need it.
• is this the only tutorial available: https://medium.com/@john_8166/praeco-walkthrough-5aada7e078a9

Thank you very much.

[nsano-rururu]

@priamai

if this is not mantained anymore servercentral / elastalert, does it mean that we will not have any future feature releases but just maybe ES compatibility releases?

servercentral / elastalert was what Praeco maintainer John Susek did at work at his previous company. servercentral / elastalert forked servercentral / elastalert johnsusek/elastalert-server because John Susek no longer has administrator privileges. Now I'm moving to another company, and I'm a co-maintainer for maintenance because I can't spend time on that work, including Praeco. Will it be maintained in the future? I don't know It's not something you have to promise. I'm not obsessed with you, and I'm not getting paid for this maintenance. First of all, don't you understand what OSS is like?

About OSS

Manuals may not be organized
The developer is not responsible for fixing the bug
In many cases, the support of the developer cannot be received

for the table praecoapp/elast-server the tickbox for ES7 is not enabled does it mean has not been tested extensively? Do you have some unit tests that you run to make sure the generated ruels are working? I could help with that since I need it.

I don't know what you're talking about. What are ES7 checkboxes? .. Unit tests haven't been done since bitsensor

is this the only tutorial available: https://medium.com/@john_8166/praeco-walkthrough-5aada7e078a9

https://medium.com/@ibrahim.ayadhi/hello-and-welcome-to-our-new-article-which-will-be-covering-the-alerting-part-in-our-socaas-136cf6258c49
https://designetwork.daichi703n.com/entry/2019/03/24/Praeco-ElastAlert-GUI
https://designetwork.daichi703n.com/entry/2020/02/10/praeco-elastalert-issue-with-es7
https://designetwork.daichi703n.com/entry/2020/02/24/praeco-helm-kubernetes

[nsano-rururu]

@priamai

I think it is doubtful to understand the relationship between the elasticsearch-py version and the elasticsearch version, so I will explain it.

# Elasticsearch 7.x
elasticsearch>=7.0.0,<8.0.0

# Elasticsearch 6.x
elasticsearch>=6.0.0,<7.0.0

# Elasticsearch 5.x
elasticsearch>=5.0.0,<6.0.0

# Elasticsearch 2.x
elasticsearch>=2.0.0,<3.0.0

See the comments below for the reason why elasticsearch == 7.0.0 is set.
#2593 (comment)

[nsano-rururu]

@priamai

Do you understand this figure first?
https://user-images.githubusercontent.com/611996/52892144-90a19300-3155-11e9-8050-cb4a440411a4.png