Home

Welcome to the BITS Darshini wiki!

This page serves as a landing page for the wiki and provides information about how to build and deploy the application. For more information about the components of this application, please check other wiki pages.

Vagrant based Installation

Prerequisites

To run a vagrant based instance of BITS-Darshini, please make sure you have vagrant and virtualbox installed on your system.

Copy code base to a separate installation directory

> git clone https://github.com/prasadtalasila/BITS-Darshini.git -b dev
> mkdir darshini-install
> cp -r BITS-Darshini/* darshini-install/
> cd darshini-install

Setting up Vagrant

Open a shell, cd into the root of the BITS-Darshini repository, and type the following command:

> vagrant up

It will typically take an hour or so for the vagrant up to complete, depending on your network performance and will also consume a high amount of data. It is very important that the provisioning process complete successfully before trying to interact with your instance: some things might superficially seem to work on a partially configured machine, but it is almost guaranteed that some things will break in hard-to-diagnose ways if vagrant provision is not run to completion.

> vagrant ssh

This will give you access to vagrant machine running the BITS-Darshini instance. cd darshini will take you to the working directory.

Working with the instance

To build BITS-Darshini, ssh into the vagrant machine and use the following commands:

> mvn clean package
> bash scripts/redeploy.sh

The application can be accessed from your browser on the url -

http://localhost:8080/protocolanalyzer/

Make sure no other application is running on port 8080.

The other instructions like (re)starting the server, checking the logs etc. are similar to manual build instructions given below.

Halting vagrant

When the application is not in use, halt the vagrant machine so that unnecessary resource consumption doesn't happen.

> exit
> vagrant halt

Restarting vagrant

To restart the machine and use the application, use the following commands:

> vagrant up
> vagrant ssh

The application can once again be accessed from your browser.

Note that vagrant up won't take as much time to complete as it did the first time.

Installing other tools

There are more stable and popular tools that does packet parsing and analysis. We consider bro IDS, ntopng and tshark for comparison of performance.

To install these tools, run

> vagrant ssh
> sudo bash scripts/tool-install.sh

During installation, you will asked to choose options for saving ipv4 and ipv6 tables (by iptables-persistence package) and allowing non-sudoers to capture packets (by dumpcap package). Choose all these options as YES.

At the end, logout of vagrant machine and login again to make the settings take effect.

> exit
> vagrant ssh

Host Installation

This scenario covers direct installation of BITS Darshini on the host operating system; we do not use the virtualization technologies during this type of installation.

Copy code base to a separate installation directory

> git clone https://github.com/prasadtalasila/BITS-Darshini.git -b dev
> mkdir darshini-install
> cp -r BITS-Darshini/* darshini-install/
> cd darshini-install

Install the software, comparable tools and their dependencies using the command:

> sudo bash scripts/install.sh
> sudo bash scripts/tool-install.sh

Export PATH environment variable for broIDS

> echo "export PATH=$PATH:/usr/local/bro/bin">> ~/.bashrc
> source ~/.bashrc

To build BITS-Darshini, run the following maven commands:

> mvn clean package
> bash scripts/redeploy.sh

The application can be accessed from your browser on the url -

http://localhost:8080/protocolanalyzer/

Make sure no other application is running on port 8080.

The other instructions like (re)starting the server, checking the logs etc. are similar to manual build instructions given below.

Manual Build

Step-by-step instructions for manual build of BITS Darshini

This is a Maven application and hence can be built with maven. Make sure you have Maven installed on you system.

Command line

From the root directory of the application (where pom.xml file is present) type the following command in your terminal -

mvn package

For the very first time, it can take a few minutes where it has to download the transitive cover of the dependencies mentioned in pom.xml file. Subsequent builds should be faster.

You can even make a clean build in subsequent times by following command -

mvn clean package

Make sure you have Maven-clean plugin installed.

IDE Plugins

If you are using any IDE like Eclipse then make sure you have the Maven Plugin installed (e.g. Eclipse-Maven-Plugin).

Deploy

Installation

Tomcat7 is used as a container for loading servlets for this application. To install tomcat7 via Apt package manager run the following command in your terminal -

sudo apt-get install tomcat7 tomcat7-admin tomcat7-common

This will install tomcat7 as a service which can be started with -

sudo service tomcat7 start

Pre-Deployment Steps

Before you can deploy the war file for this application on Tomcat, you have to configure Tomcat correctly. To do so please follow the guide below.

Directory Structure

• CATALINA_HOME - By default, this environmental variable will be set to point to the /usr/share/tomcat7/ directory. This directory contains the bin folder among other folders which holds all the tomcat related scripts, most important of which is catalina.sh script. This scripts starts Tomcat and runs applications within it.

• CATALINA_BASE - Although there is no default value for this environmental variable, it should point to the directory which contains folders like conf, webapps, logs, server etc. If you have installed Tomcat via apt-get then you can find these folders in /var/lib/tomcat7 directory.

Roles and User Credentials

To access manager webapp through browser or deploy applications with command line tools, one has to define appropriate manager-roles in the tomcat-users.xml file present in /var/lib/tomcat7/conf/ directory. Some example roles and user credentials are mentioned below -

<role rolename="manager-gui"/>
<role rolename="manager-script"/>
<user username="admin" password="admin" roles="manager-gui" /> 
<user username="adminscript" password="passwordscript" roles="manager-script" />

Now one can deploy application war files through either text/command line interface or Manager web-app. Of course if you want to fine-tune Tomcat further, you can refer to official Tomcat7 documentation. One such useful configuration is mentioned on this page (See Appendix B) for avoiding out of memory issue.

For this particular application, only one more step of configuring elasticsearch server is remaining before you can deploy it on Tomcat server. Check out the next section for it.

Elasticsearch Properties Configuration

Spring servlet starts an integrated Elasticsearch server instance at the time of application loading. Elasticsearch server is being configured (as of now) by reading its configuration properties from elasticsearch.properties file. Elasticsearch needs two directories for storing data and logs respectively. Paths to those directories are specified in elasticsearch.path.data and elasticsearch.path.logs properties. Currently they point to a tmp directory where everyone has write access. But you should change the path to point to appropriate directory where only system user "tomcat7" has write permissions. Make the appropriate changes and save the file.

Once properties are configured, rebuild the project with Maven. Now the application is ready to be deployed on Tomcat.

Deployment

Maven tomcat7 plugin

Since manual deployment is error-prone, it is advised that user should use a tomcat plugin (to whatever application that is handling this project. E.g. Jenkins or Maven) for hassle-free deployment. Maven tomcat7 plugin is one such excellent contender.

First make sure tomcat is running by running following command -

sudo service tomcat7 status

If not then -

sudo service tomcat7 start

Discover Maven-home by typing -

mvn --version

where it will show the Maven home path in the first line of the output.

Go to <Maven-Home>/conf/ directory and open settings.xml file for writing (root owned file!). Find the section for servers tagged with <servers> tag. An example <server> tag will be given. After the comment line, add following xml configuration -

<server>
    <id>TomcatServer</id>
    <username>adminscript</username>
    <password>passwordscript</password>
</server>

where the user credentials provided must match with those for the role of manager-script role in tomcat-users.xml file. If you wish to change server id, then the corresponding change must be reflected here as well. Save this file.

Then make sure you are in the root folder for this project (where pom.xml file resides) and type in terminal -

mvn tomcat7:deploy

The application should get deployed on the url -

http://localhost:8080/protocolanalyzer/

NOTE - For the first time everything should work by following the instructions above. After subsequent builds with maven, one has to run -

mvn tomcat7:redeploy
sudo service tomcat7 restart

to successfully redeploy the application on the server.

Debugging and Status Checking

If all the steps mentioned above are carefully followed then the deployment should succeed. However in case of any failures or errors one can look at the log files present in -

/var/lib/tomcat7/logs/

directory. Server-wide logs are stored in catalina.<DATE>.log file. Host-wide (e.g. localhost applications) logs are stored in localhost.<DATE>.log file. INFO logs will be present in packetanalyzer.log file. One or the other log file will lead you to the root cause of the error.

Appendix A - Manual Deployment on Tomcat7

Note that manual deployment of WAR file on Tomcat is fraught with difficulties and errors. You are strongly advised to use other approaches (like maven-tomcat7 plugin) for deployment. Only in case of failure in former case, you should turn to this guide.

Out of many approaches for manually deploying a WAR file, following is the simplest and least error-prone. WAR file deployment on Tomcat can be achieved in two ways -

Command line - Curl

curl --upload-file /path/to/war/file/protocolanalyzer-1.0-SNAPSHOT.war "http://adminscript:adminscript@localhost:8080/manager/text/deploy?path=/protocolanalyzer&update=true"

where /path/to/war/file should be replaced appropriately. Be careful to provide appropriate user credentials for the role of manager-script.

Manager Webapp

Open manager webapp from browser by hitting following URL -

http://localhost:8080/manager/

(You will have to provide appropriate user credentials saved in tomcat-users.xml file for the role of manager-gui)

In the manger webapp you can specify the context path (e.g. /protocolanalyzer) and upload a WAR file to be deployed.

After a successful deployment, the application can be accessed on following URL -

http://localhost:8080/protocolanalyzer/

Installing other tools

There are more stable and popular tools that does packet parsing and analysis. We consider bro IDS, ntopng and tshark for comparison of performance.

To install these tools, run

> vagrant ssh
> sudo bash scripts/tool-install.sh

During installation, you will asked to choose options for saving ipv4 and ipv6 tables (by iptables-persistence package) and allowing non-sudoers to capture packets (by dumpcap package). Choose all these options as YES.

Export PATH environment variable for broIDS

> echo "export PATH=$PATH:/usr/local/bro/bin">> ~/.bashrc
> source ~/.bashrc

Appendix B - Tomcat7 Java Heap Space Exceeded Issue

When Tomcat is run as a service, by default it is configured to run with very little java space (128MB to be precise! Check out JAVA_OPTS in /etc/default/tomcat7 file..)

If for the large pcap files, your experiment is throwing out of memory (or java heap space error) then you can increase the default threshold by overriding it. Don't alter the aforementioned file directly. The recommended way to override tomcat default configurations is as follows -

• Open a file named setenv.sh in $CATALINA_HOME/bin/ directory ($CATALINA_HOME would usually be /usr/share/tomcat7)
• Add the line similar to the following -

CATALINA_OPTS="$CATALINA_OPTS -server -Xms512m -Xmx2048m"

where "Xmx" is the maximum amount of memory you want to grant to Tomcat. Make sure whatever number you fill in, your hardware is capable of supporting it (accounting for other processes running as well)