-
Notifications
You must be signed in to change notification settings - Fork 0
4.0 Setting Up a System
The Basics (Boot Strapping)
Flamingo is Client-Server, with ‘pond’ being the server, and ‘flamingo’ being the client.
One server can manage flows from many flow generators (routers), and one server can also manage many clients. One client can connect (one at a time) to many servers. It is handy to have a client on the same machine as the server for testing, or in a simple configuration, this one client may be all that is needed.
The server (pond) can accept live data directly from a router without the data being archived. This is probably the best way to start.
NetFlow and sFlow data is sent from the router to the flow collector using UDP, the client and server are connected together with TCP.
Pond uses a configuration file named pond.conf. It will search for it in the current directory, flamingo/data and also other directories. An example pond.conf is included in flamingo/data/pond.conf.sample. Copy this file to pond.conf and launch pond, and lets see if we can get pond to find it.
Once pond seems to be running happily, launch flamingo under X-Windows and click the “Connect” button. In the connect window, enter the IP address of your server (or 127.0.0.1 if both flamingo and pond are running on the same server) and with the sample config, file the port for clients to connect is 4444 and the default password is “flamingo” (no quotes).
Now at the bottom of the Flamingo window it should say something like “Connected to 127.0.0.1:4444 – 7 available feed(s)”
Now lets try to get a Cisco router sending NetFlow to the server. On your router type;
Config t
Ip flow-export version 5
Ip flow-export destination My-Server-IP-Address 20655
(Where 20655 is the UDP port on the server where pond is listening for packets from a router it calls AA1, according to pond.conf.sample)
Now we need to add NetFlow to one or more interfaces (lets use interface gi0/1 for this example)
(in config t mode still, type;)
Int gi0/1
Ip flow ingress
Ip flow egress
Exit
Exit
Now, as long as your firewall is not blocking the flow data we should start to get something on the flamingo client. As soon as you do get something on the client, you may want to skip to the [Using Flamingo](5.0 Using Flamingo) section, and return here after you are more familiar with the use of the client.
For the simplest of setups, (One Cisco router and no archive) this may be all you need.
Troubleshooting the basic client setup
On the client next to the “Connect Button” icon under which is most likely the text “Src IP” This is the current view. Just to the right of this icon is a down arrow which is a drop down menu. Click this down arrow and select different view such as “Src/Dst IP”
On the next icon to the right, make sure we have selected router AA1
On the next icon to the right, change from 10 seconds to 1 second.
On the next icon to the right, change from Aggregated to Not Aggregated.
Click the “Show Controls” button.
In the Info window, do we have any data yet?
On the “Controls” window, scroll down with the elevator and at the bottom put a check for “Enable Lables”.
Back on the flamingo window click the Preferences button.
On the “Preferences” window select the “Ranges” tab.
Ensure you have a check for both a “Source Range” of 0.0.0.0 0 and a”Destination Range” of 0.0.0.0 0
Close the preferences window with “OK”
If you still have no data in the flamingo display area, you may not have any data crossing the specified ports on the router, or you may have a firewall issue.
When you do get something (Anything) in the way of data on Flamingo, review [Using Flamingo](5.0 Using Flamingo) and pick back up here.
For the simplest of setups, (One Cisco router and no archive) this may be all you need.
Setting up an archive of flow data
To archive flow data we will set two software “shims” from flow-tools, flow-fanout, and flow-capture. These programs should already be on your server as the flow-tools package was required to compile Pond.
Flow-fanout will be used to accept flow data from our Cisco router and send one copy to pond for live updates, and send a second copy to flow-capture for archiving.
These processes will communicate with each other using different UDP ports. In our example above, we have our router sending to our server on UDP port 20655. And we have pond listening for flow data on UDP port 20655 as defined in the example pond.conf. We will need to get this flow data passing through flow-fanout.
Lets assume that we would like to use UDP port 20012 for flow-fanout to receive flow data from the router, and we would like copies sent to pond on 20655 and slow to flow-capture on port 20013.
We could start flow-fanout like this;
Flow-fanout –p - -V 5 (My-Server-IP-Address)/0.0.0.0/20012 0.0.0.0/(My-Server-IP-Address)/20013 0.0.0.0/(My-Server-IP-Address)/20655 &
(Where My-Server-IP-Address gets replaced with the IP address of your server.)
Next we can change our router so that it will send to flow-fanout instead of directly to pond.
On the router;
Config t
No Ip flow-export destination My-Server-IP-Address 20655
Ip flow-export destination My-Server-IP-Address 20012
We should still be able to view live updates on Flamingo, only now they flow from the router, through flow-fanout, through pond, all of the way to flamingo.
Now we are ready to launch flow-capture but the location on disk where flow-capture places the archive, must match pond.conf in order for the server to find the archives.
In the example pond.conf the Netflow-Archive_Path is set for ./nfbin/ which is probably a bit too variable. We should probably use something more like /usr/nfbin/ which could also make a nice mount point for dedicating a whole disk drive to storing this data.
We could launch a flow capture something like this;
Flow-capture –D –n 95 –c 15Min –e 9600 –C 100days –N 3 –V 5 –w /usr/nfbin/AA1 0/My-Server-IP-Address/20013 &
If all went well, in about 15 minutes or in some cases, due to a bug, an hour and 15 minutes, we should be able to play back an archived event for router AA1 with the Flamingo client.
On the client click the down arrow to the right of the third icon (Feed selection) and choose “Playback Feed”. Choose a reasonable time and date for the playback to begin. The text on the main flamingo window, bottom, right, should change from “Live Update …” to “Playback Update” followed by the date and time of the playback. You should also observe carefully what time is being displayed, due to a bug, it’s some times a hour off of what you asked for.
Setting up to receive sFlow streams and convert them to NetFlow
For this task we’ll use an open source tool called sFlow Toolkit available from inMon.
http://www.inmon.com/technology/sflowTools.php
You will need to download the source and compile it.
We’ll use it much the same way as we used flow-fanout above.
Let’s suppose that you had completed the setup above and wanted to replace the Cisco router with a Brocade router.
We last had our Cisco router sending flow data to My-Server-IP-Address on UDP port 20012. So, lets start sflowtool to listen on UDP port 20011 and send the converted flow data to UDP port 20012 (Our flow Flow-fanout process we started above in “Setting up an archive of flow data”)
We would start sflowtool like this;
Sflowtool –p 20011 -4 –c My-Server-IP-Address –d 20012 &
Perhaps you don’t have a Cisco router, only a Brocade router and have not made it as far in the setup as archiving. In this case you may want to start out sending the flow data from sflowtool directly to pond.
In the “Boot Strapping” section at the top of this page, we use the example pond.conf which has a router AA1 defined as UDP port 20655. We can have sflowtool send the converted flow data directly to pond like this;
Sflowtool –p 20011 -4 –c My-Server-IP-Address –d 20655 &
On the Brocade router we would add something like the following to the global config;
Sflow enable
Sflow My-Server-IP-Address 20011
Sflow sample 4096 (or any other sampling rate you wanted)
And in the interface configuration of the interfaces we want to collect data from we would add;
Sflow forwarding
Setting up a multi-router, multi-vendor, archiving, enterprise system, using the script “Fly”
With as many as 4 UDP ports assigned to each flow generator (router) this could quickly become a nightmare to keep track of with just a handful of router.
To make things easier we have a script which you may want to use to launch all of the various processes and assign appropriate UDP port numbers.
The script below should be fairly self explanatory:
#!/usr/local/bin/bash
#
# This script will killall flamingo, pond, sflowtool, flow-capture, flow-fan, and ping processes
# and restart them. M. W. Gemeny, USMD/MDREN
#
# Version V1.1 07/01/15
#
# Please set these to match your system
#
myip='192.168.1.3'
mygateway='192.168.1.1'
Netflow_Archive_Path="/usr/nfbin/" # must match pond.conf but with double quotes here, not there
#
# The next set of variables will get redefined later and are shown here for reference
#
SrcName="MyRouterName" # Must match pond.conf but with double quotes here and [] there
instance='00'
sflow='no'
capture='no'
#
# Make sure this is safe to do on your system!
#
echo
echo
#killall -c flamingo
#killall -c pond
#killall -c sflowtool
#killall -c flow-capture
#killall -c flow-fanout
#killall -c ping
#killall -c x11vnc
#
#
# Note that;
# One instance of Flamingo's pond server can listen on many UDP ports as configured in pond.conf
# and can support many Flamingo clients and playback from many archives.
# One instance of sflowtool can only listen on one UDP port, but can log, and convert, and forward to many ports.
# One instance of flow-fan can only listen on one UDP port, but can forward to many ports and/or IP addresses,
# and can not do nothing else.
# One instance of flow-capture can only listen on one UDP port and can create only one disk archive.
#
# A system may look like this;
#
# Netflow Generator Sflow-Generator (nn)
# | |
# | | |--------> Legacy Collector A
# | | |--------> Legacy Collector B
# | V (UDP 20nn1) |
# | sflowtool ^---------------
# | | ^
# -------------- | |
# | | |
# V V (UDP 20nn2) |
# flow-fanout -------------------------------------> flow-fanout (Aggregator)
# | | (UDP 20000) |
# | | |
# | | |
# V (UDP 20nn3) | V (UDP 20001)
# flow-capture --------------> (Flamingo) pond
# | (UDP 20nn4) | | | |
# | / | | |
# | / | | |
# V / | | | (TCP)
# (disk archive) <--->>>--------------------------- | | |
# | | V
# | V flamingo (Client)
# V flamingo (Client)
# flamingo (Client)
# ...
#
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^
# Duplicated for each flow generator One instance or more as required
#
#
# First, let's define a subroutine to launch a set of processes for a flow generator such as described above/left
#
#echo test point one
launch_processes () (
echo Starting damons for "$SrcName" instance "$instance"
# only start the sflow converter if required
if [ "$sflow" = "yes" ]; then
echo Starting sflowtool for "$SrcName"
sflowtool -p 20"$instance"1 -4 -c "$myip" -d 20"$instance"2 &
fi
echo Starting flow-fanout
flow-fanout -p - -V 5 "$myip"/0.0.0.0/20"$instance"2 0.0.0.0/"$myip"/20000 0.0.0.0/"$myip"/20"$instance"3 0.0.0.0/"$myip"/20"$instance"4 &
#
if [ "$capture" = "yes" ] && [ -d "$Netflow_Archive_Path" ] && ! [ -d "$Netflow_Archive_Path""$SrcName" ]; then
echo Creating "$Netflow_Archive_Path""$SrcName"
mkdir "$Netflow_Archive_Path""$SrcName"
else
echo No need to create archive path for "$SrcName"
fi
#
# only start the capture if required
if [ "$capture" = "yes" ] && [ -d "$Netflow_Archive_Path" ]; then
echo Launching flow-capture for "$SrcName"
flow-capture -D -n 95 -C 15Min -e 9600 -C 100days -N 3 -V 5 -w "$Netflow_Archive_Path""$SrcName" 0/"$myip"/20"$instance"3 &
fi
)
#
# Now with that defined, lets start getting things done!
#
#
#
# We have found that many routers and switches will not ARP for only UDP packets
# As such we find the flow data gets broadcast to otherwise quiet collectors
# One solution is a slow ping to the default gateway
#
echo Starting ping
ping -i 20 "$mygateway" > /dev/null&
#
# Now start damons for single instances (Global)
#
#First the Aggregator (this is where you would add pointers to your legacy collectors)
echo Starting flow-fanout for aggregator
flow-fanout -p - -V 5 "$myip"/0.0.0.0/20000 0.0.0.0/"$myip"/20001 &
#
#And then pond
# Comment out one of the next two lines based of debuging messages or no debugging messages
#G_MESSAGES_DEBUG=all pond &
G_MESSAGES_DEBUG=none pond &
#pond &
#
#
# Now let someone know if the Netflow Archive Path does not exist
#
if [ -d "$Netflow_Archive_Path" ]; then
echo Main Archive Path found
else
echo
echo Oops, the Netflow Archive Path does not exist!
echo With pwd of "$PWD", "$Netflow_Archive_Path" can not be found.
echo No capturing will be done and no population of the
echo tree will be done!
echo
echo Please create "$Netflow_Archive_Path" if you are serious about archiving here!
echo If you do that and run this again we will populate the tree and start loging for you.
echo
echo
fi
#
# Then start Damons for each flow generator
# Note that these also need entries in pond.conf to create the port 20nn4 listeners
# Also the SrcName used here needs to match the source name in pond.conf in order for
# Flamingo's pond server to find the archive created by flow-capture to play it back
# for the client when requested. Add as many of these as you need, one for each flow source.
#
#UMSA-GW (nn=01)
SrcName="UMSA-GW" # Must match pond.conf but with double quotes here and [] there
instance='01'
sflow='no'
capture='yes'
launch_processes
#
#UMCP-CORE (nn=02)
SrcName="UMCP" # Must match pond.conf but with double quotes here and [] there
instance='02'
sflow='yes'
capture='yes'
launch_processes
#
#CES-LESREC (nn=03)
SrcName="CES-LESREC" # Must match pond.conf but with double quotes here and [] there
instance='03'
sflow='no'
capture='yes'
launch_processes
#
# Thats it for the for generators
# Now do the final cleanup and launch the client
# Unless this is a server only device then comment this out
#
#echo And last start the user interface for flamingo
# Comment out all but one of the next three lines based on debuging messages or no debugging messages
#G_MESSAGES_DEBUG=all flamingo &
G_MESSAGES_DEBUG=none flamingo &
#flamingo &
#
# You may also want to start a VNC server
x11vnc -shared -alwaysshared -loop -passwd 123456 &
#
# You may also want to turn off the screen saver
xset s off
#
#
echo @fin
echo