Tested with Docker version v27.0.3
and Docker Compose version v2.29.1
.
Backend with fastapi+uvicorn for log analysis with LLMs and MySQL queries.
- Log Analyzer with LLMs and MySQL
Create a .env
file with the following keys with updated values for unames and pass:
# set to ERROR for deployment
DEBUG_LEVEL=DEBUG
# http api server
API_SERVER_PORT=8080
# openai api key
OPENAI_API_KEY=<OPENAI_API_KEY>
# langchain langsmith keys
USER_AGENT=log_analyzer
LANGCHAIN_PROJECT=log_analyzer
LANGCHAIN_TRACING_V2=true
LANGCHAIN_ENDPOINT=https://api.smith.langchain.com
LANGCHAIN_API_KEY=<LANGCHAIN_API_KEY>
# mysql mariadb
MYSQL_HOST=mysql
MYSQL_PORT=3306
MYSQL_USER=user
MYSQL_PASSWORD=pass
MYSQL_DATABASE=default
MYSQL_ROOT_PASSWORD=admin
# phpmyadmin mariadb
PMA_GUI_PORT=8001
PMA_HOST=${MYSQL_HOST}
PMA_PORT=${MYSQL_PORT}
PMA_USER=${MYSQL_USER}
PMA_PASSWORD=${MYSQL_PASSWORD}
mkdir -p volumes/log_analyzer
mkdir -p volumes/store
There are two options for running the analysis service. Both require docker compose
(Available from the official docker site). $docker-compose ...
style commands have been depreciated.
Note: some services are set to bind to all addresses which should be changed in a production environment.
# build all required containers
docker compose build
# start all services
docker compose up -d
The server will be available at http://localhost:8080 if using the default port.
When changing settings in docker-compose.yaml
for the mongodb service, the existing docker and shared volumes might have to be purged i.e. when changing replicaset name.
WARNING: This will delete all existing user, document, and vector records.
docker-compose down
docker volume rm $(docker volume ls -q)
rm -rf volumes/store
# build all required containers
docker compose build
# start mysql server & phpmyadmin server
docker compose up -d mysql mysql-admin
Build server container and start server at HTTP port EXPOSED_HTTP_PORT
bash scripts/build_docker.sh
bash scripts/run_docker.sh -p EXPOSED_HTTP_PORT
The server will be available at http://localhost:8080 if using the default port.
Install requirements inside venv or conda environment
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
Start server at HTTP port EXPOSED_HTTP_PORT. Note the host names must contain addresses when using docker microservices and the fastapi+uvicorn server outside the docker compose environment.
python app/server.py -p EXPOSED_HTTP_PORT
The server will be available at http://localhost:8080 if using the default port.
pip install streamlit==1.38.0
streamlit run app/streamlit_frontend.py
WARNING: Never use for production
# start log analyzer with python
# sign up for ngrok account at https://ngrok.com/
# https://ngrok.com/docs/using-ngrok-with/docker/
docker pull ngrok/ngrok
# for linux systems
docker run --net=host -it -e NGROK_AUTHTOKEN=<NGROK_AUTHTOKEN> ngrok/ngrok:latest http <EXPOSED_HTTP_PORT>
# for MacOS and windows
docker run -it -e NGROK_AUTHTOKEN=<NGROK_AUTHTOKEN> ngrok/ngrok:latest http host.docker.internal:<EXPOSED_HTTP_PORT>
Note: all the microservices must already be running with docker compose.
Install requirements:
pip install -r tests/requirements.txt
Run tests:
pytest tests/
Generating coverage reports
coverage run -m pytest tests/
coverage report -m -i
The new SQL table should also be created through the PHPMyAdmin GUI/mysql command line inside the mariadb container.
The following files must be edited.
- Edit
app/static/sql/init.sql
for changing/adding log table schema - Edit
app/models/model.py
to add/edit the LogFileType - Edit
app/api/log_format/log_parser.py
for parsing logs - Edit
app/core/setup.py
for adding table schema and data sample info for text2sql conversion
Editing Tests
- Edit
tests/conftests.py
for setting the correct values for the test database - Edit
tests/api/test_mysql_api.py
for setting the correct values for the test database