Anomaly-Flow

Framework for implementing Intrusion Detection Systems (NIDS) aimed at identifying anomalies in network flows using Machine Learning models.

Create the Experimental Environment

To create the experimental environment, execute the following commands:

python -m venv .env 

Note: The virtual environment must have this name because the scripts use this name to load the needed libraries. fl-unsup-nids/blob/anomaly-flow-baseline/utils/load_data.py After creating the Virtual environment, use the following command to install the necessary python packages:

pip install -r requirements.txt

Commands to run local experiments

To run the local experiments, use the following command:

source experiments.sh

Commands to run the Federated Learning experiments

To run the Federated Learning experiments, use the following command:

source run.sh 

Getting the data to run the experiments

To run the experiments, we need to have the CIC-IDS2018, BoT-IoT and ToN-IoT in a folder called datasets in the root of the project, e.g. :

|- anomaly-flow
    |- datasets 
        |- NF-CSE-CIC-IDS2018-v2-DDoS.csv.gz
        |- NF-BoT-IoT-v2-DDoS.csv.gz
        |- NF-ToN-IoT-v2-DDoS.csv.gz 

In our case we used derived datasets containing only benign and DDoS samples filtered from the originals datasets.

Generating Synthetic Data

To generate Synthetic Data for a specific dataset use the script main.ipynb.

External models Simple Models Baseline

Install the EFC package

To run the external and simple model scripts, please install the custom pip package for the EFC algorithm:

1. Make sure the scripts have run privileges:

chmod +x ./auxiliary_scripts/install-efc.sh

2. Run the script (Git and the anomaly-flow environment named .env required):

./auxiliary_scripts/install-efc.sh

Baselines

The baselines for a multi-domain DDoS detection using federated learning are available here:

• Autoencoder
• Logistic Regression
• FLAD

References

The data used to carry out the experiments can be obtained from:

Machine Learning-Based NIDS Datasets (Netflow V2 Datasets)

The following projects were used as reference for the preparation of this work:

GANomaly

Anomaly-Toolbox Project

Energy-based Flow Classifier

Authors

   Leonardo Henrique de Melo

   Gustavo de Carvalho Bertoli

   Michele Nogueira

   Aldri Luiz dos Santos

   Lourenço Alves Pereira Junior

Cite this work