GreyNoise Visualization Application

A simple web application built to visualize GreyNoise data. Includes some simple statistics (general stats and time series charts), a table view of the data, and a map to view the general location of the IP addresses that are associated with a particular tag. The frontend is built with Angular and retrieves data from the Flask backend that uses Redis for caching. The Flask backend retrieves data from GreyNoise (with caching), computes statistics, and uses the MaxMind database for geolocation of IP addresses.

Excerpt from the GreyNoise repo:

GreyNoise is a system that collects and analyzes data on Internet-wide scanners. GreyNoise collects data on benign scanners such as Shodan.io, as well as malicious actors like SSH and telnet worms.

Table of Contents

Getting Started
Development
Testing
Deployment
License
Acknowledgments

Getting Started

The quickest way deploy this project is by using docker-compose; however, in order to set up a development environment, follow the steps in the development section.

Prerequisites

For development:

• Python 3.6
• GeoLite2 City database
• Node.js
• Angular CLI - npm install -g @angular/cli

For deployment:

Install Docker and Docker Compose:

• Docker installation instructions: https://docs.docker.com/engine/installation/

curl -fsSL get.docker.com -o get-docker.sh
sudo sh get-docker.sh

• Docker Compose installation instructions: https://docs.docker.com/compose/install/

sudo curl -L https://github.com/docker/compose/releases/download/1.18.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version

Development

Flask backend:

cd backend
pip3.6 install -r requirements.txt
wget "geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz"
tar -xvf GeoLite2-City.tar.gz
mv GeoLite2-City*/GeoLite2-City.mmdb .
python3.6 app.py

Angular development server:

cd frontend
npm install
ng serve

Browse to http://localhost:4200 to view the application

Testing

Angular:

Unit testing:

cd frontend
ng test 

e2e testing:

Either requires the Flask backend to be running locally as shown in the development section or the environment flag can be passed with the "e2e" option to use the containerized Flask backend (docker-compose).

cd frontend
ng e2e

OR

ng e2e --environment=e2e #sets the api url to http://localhost 

Flask:

cd backend
nose2

Deployment

A simple deployment using docker-compose:

There will be some warnings about unset variables; however, they will not cause any issues if SSL is not being used. You can simply ignore these warnings, or set the variables to blank strings.

The application will be accessible at http://localhost.

docker-compose build
docker-compose up -d

In order to set up NGINX to use SSL:

The SSL cert and key are set to be in /etc/ssl/certs/greynoise/greynoise.crt and /etc/ssl/certs/greynoise/greynoise.key on the host machine running the containers. See optional SSL configuration below to configure cert names/paths.

Also, the lines 34-36 in docker-compose.yml need to be uncommented or altered based on the cert paths of the host machine.

export GREYNOISE_NGINX_SSL=true #will enable SSL
export GREYNOISE_SERVER_NAME=localhost #or whatever the server name will be
docker-compose build
docker-compose up -d

Optional SSL configuration:

The cert/key paths can be changed in the docker-compose.yml file on line 33 and the cert/key names can be changed by setting the GREYNOISE_CERT_NAME environment variable.

#sets NGINX to look for /etc/ssl/certs/greynoise/test.crt and /etc/ssl/certs/greynoise/test.key
#unless the paths were changed in the docker-compose file
export GREYNOISE_CERT_NAME=test

License

This project is licensed under the MIT License - see the LICENSE file for details

Acknowledgments

• GreyNoise API
• Time series function