Skip to content

Commit

Permalink
Merge pull request #1 from mami-project/cloud_init_dev
Browse files Browse the repository at this point in the history 
Merged Cloud_init_dev to master
  • Loading branch information
@buehlert
buehlert authored Oct 9, 2018
2 parents 8335585 + 4e7b69a commit 163cdd3
Show file tree
Hide file tree
Showing 22 changed files with 2,546 additions and 1,379 deletions.
211 changes: 144 additions & 67 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,25 @@
# PLUS Middlebox using FD.io/VPP
# VPP-based passive latency measurement middlebox

## IETF hackathon code
Checkout branch *quic_IETF_hackathon*
This VPP plugin adds support for passive latency measurements in FD.io.
The current implementation will estimate the RTT of:

## Installation
- QUIC flows using the latency spin signal (and other techniques) described
in our IMC'18 paper [Three Bits Suffice](https://nsg.ee.ethz.ch/fileadmin/user_upload/spinbit.pdf).
The following [fork of minq](https://github.com/pietdevaere/minq) adds the latency spin signal to QUIC traffic
such that it is detectable by the VPP plugin.
- TCP flows using the latency spin signal and/or TCP timestamps.
We provide [patches](https://github.com/mami-project/three-bits-suffice/tree/master/tcp/kernel_patches)
to add latency spin signal support to the Linux kernel.
- [PLUS](https://nsg.ee.ethz.ch/fileadmin/user_upload/CNSM_2017.pdf) flows using the PSN and PSE header fields.
For example [puic-go](https://github.com/mami-project/puic-go) can be used to add PLUS support to quic-go.

## Installation

You can either use Vagrant to set up everything automatically
or compile the plugin in an existing VPP installation.
The plugin is tested with the stable FD.io version 17.10.

### Using Vagrant
If not already available, install *Vagrant* and *VirtualBox* on your machine.
Go to the Vagrant directory and execute:
```
Expand All @@ -14,7 +30,7 @@ To start Vagrant and connect via ssh (root access without password).

Part of the Vagrant setup adapted from the [vpp-mb](https://github.com/mami-project/vpp-mb) project.

## Additional Vagrant commands
### Additional Vagrant commands
Rsync the vpp-plus directory once more (e.g. useful after git pull):
```
cd vagrant
Expand All @@ -32,6 +48,18 @@ vagrant up
vagrant ssh
```

### Compiling the plugin
To compile the plugin manually or adapt changes inside Vagrant, use:
```
cd latency-plugin
sudo autoreconf -fis
sudo ./configure
make
sudo make install
```

Restart VPP, e.g. `sudo service vpp restart`

## Important VPP commands
Start VPP: `sudo service vpp start`

Expand All @@ -41,89 +69,138 @@ You can either access the VPP shell with `sudo vppctl` and then interactively ex

Use `sudo vppctl help` for a list of supported commands.

### Important general commands
### General commands
List of interfaces: `sudo vppctl show interface` (you can also shorten the commands, e.g. `sudo vppctl sh int`)

Show the VPP graph: `sudo vppctl show vlib graph`

Add packet trace for (50 packets) `sudo vppctl trace add af-packet-input 50`
Add a packet trace storing 50 packets `sudo vppctl trace add af-packet-input 50`

Display the trace: `sudo vppctl show trace`
Display the captured packets in the trace: `sudo vppctl show trace`

Execute multiple commands from a file: `sudo vppctl exec <file>`
Execute multiple VPP commands from a file (one command per line): `sudo vppctl exec <file>`

### PLUS specific commands
Add an interface to the PLUS plugin: `sudo vppctl plus <interface>`
### Latency plugin specific commands
To get an overview, use: `sudo vppctl latency help`

Remove an interface: `sudo vppctl plus <interface> disable`
Add an interface to the plugin: `sudo vppctl latency interface <interface>`

List all active PLUS flows: `sudo vppctl plus stat`
Remove an interface: `sudo vppctl latency interface <interface> disable`

## Connect Vagrant VM to host machine and run go plus-echo test
This setup assumes you use VirtualBox as provider for the Vagrant VM!
List all currently active flows with latency estimations: `sudo vppctl latency stats`

On your local machine in VirtualBox: Go to `Virtualbox --> Preferences...`. In the "Network tab" add *two* "Host-only Networks". Change the configuration:
```
Network 1: IPv4 Address: 192.168.100.1, IPv4 Network Mask: 255.255.255.0
Network 2: IPv4 Address: 192.168.101.1, IPv4 Network Mask: 255.255.255.0
```
Now start the Vagrant VM and execute the following commands *inside the VM*:
```
sudo service vpp start
sudo vppctl ex /home/vagrant/plus-mb/scripts/external_vpp_interface.conf # Add IPs to the interfaces inside VPP
sudo vppctl plus GigabitEthernet0/8/0 # Add interfaces to the plus plugin
sudo vppctl plus GigabitEthernet0/9/0
```
VPP should now be ready. Back on the *local machine*:
```
sudo route add 192.168.100.1/32 gw 192.168.101.2 # Add static routes for go client and server
sudo route add 192.168.101.1/32 gw 192.168.100.2
# if not available, install golan-1.9: sudo apt-get install golang-1.9-go
go get github.com/FMNSSun/plus-echo
cd go/src/github.com/FMNSSun/plus-echo
go build client.go
go build server.go
./server -local-addr=192.168.101.1:4000
./client -local-addr=192.168.100.1:3000 -remote-addr=192.168.101.1:4000 # in a different terminal
```
The go PLUS client should send a PLUS packet to the server and get a reply back.

In the Vagrant vm you should see two observed PlUS packets using e.g. `sudo vppctl plus stat`
Set the IPv4 address the plugin is listening to `sudo vppctl latency mb_ip <IPv4 (dot)>`

**Important:** To add a packet trace, use `sudo vppctl trace add dpdk-input 50`.
Add a UDP port number that indicates QUIC traffic `sudo vppctl latency quic_port <port>`.
Can be repeated with different ports.

## Simple example
Go to the *scripts* directory and make `ns_setup.sh` executable (`chmod +x ns_setup.sh`)
Add NAT-like functionalities `sudo vppctl latency nat <IPv4 (dot)> <port>`. This is useful if you
want to deploy the middlebox such that it can make on-path measurements taking traffic in
both directions into account. Can be repeated with different pairs of ports and IPs.
See next section for more information.

Execute `ns_setup.sh` to generate virtual namespaces veth pairs (`sudo ./ns_setup.sh`)

Start VPP: `sudo service vpp start`
## On-path latency measurements
To be able to perform on-path measurements and observing traffic from the client
to the server **and** the reverse traffic, we added NAT-like functionalities to the
latency plugin.

Execute the file `vpp_interface.conf` to connect the virtual namespaces to VPP:
As an example, assume the VPP middlebox has the IP 1.2.3.4 (defined with `sudo vppctl latency mb_ip 1.2.3.4`).
Now we would like to be able to forward traffic towards the server 5.6.7.8 through the middlebox.
For that, we arbitrarily associate port 8888 with the dst IP 5.6.7.8 and add that to the plugin with
`sudo vppctl latency 5.6.7.8 8888`. Any client can now send traffic to 5.6.7.8 (over the middlebox)
by sending traffic towards the IP of the middlebox (1.2.3.4) with dst port 8888.
Whenever the plugin receives traffic with dst port 8888, it will:

`sudo vppctl exec /home/vagrant/plus-mb/scripts/vpp_interface.conf`
1. save the observed src IP and replace it with its own IP (1.2.3.4)
1. replace the dst IP with the IP of the corresponding server (5.6.7.8)
1. send the traffic towards the new destination (src and dst ports are not changed)

(Use `sudo vppctl sh int` to confirm that the new interfaces are visible: *host-vpp1* and *host-vpp2*)
Once it receives traffic back from the server, it reverses the process and sends it to the original client.

Add the two interfaces to the PLUS plugin, such that it analyzes traffic coming from these interfaces:
Following a list of sample commands to configure VPP and the plugin to implement the previous example.
We assume that the server running VPP is inside a network with IP space 1.2.3.0/24 and the gateway
towards the Internet has the IP 1.2.3.1. The VPP server has one interface (called `GigabitEthernet3/0/0`).
The actual interface name depends one the used implementation/hardware and can be found with `sudo vppctl sh int`.

`sudo vppctl plus host-vpp1` and `sudo vppctl plus host-vpp2`

Use the (very basic) Python scripts to generate PLUS traffic. The sender and receiver are each executed in a separate namespace (connected via VPP).
```
set int state GigabitEthernet3/0/0 up
set int ip address GigabitEthernet3/0/0 1.2.3.4/24
ip route add 0.0.0.0/0 via 1.2.3.1 GigabitEthernet3/0/0
latency interface GigabitEthernet3/0/0
latency mb_ip 1.2.3.4
latency nat 5.6.7.8 8888
latency quic_port 8888
```

Receiver: `sudo ip netns exec vpp2 python receiver.py`
The last command declares traffic towards/from port 8888 as QUIC traffic.
All these commands can be saved in a file (e.g. `setup.conf`) and executed
with `sudo vppctl exec setup.conf`.

Sender: `sudo ip netns exec vpp1 python sender.py`
### Connect Vagrant to host machine

Use `sudo vppctl plus stat` to see the generated flow or use the packet trace commands from above.
If you use the Vagrant installation and want to connect the VM to the host machine,
use "Host-only Networks" (assuming VirtualBox as provider for the Vagrant VM).
On your local machine in VirtualBox: Go to `Virtualbox --> Preferences...`.
In the "Network tab" add two "Host-only Networks" and change the configuration:
```
Network 1: IPv4 Address: 192.168.100.1, IPv4 Network Mask: 255.255.255.0
Network 2: IPv4 Address: 192.168.101.1, IPv4 Network Mask: 255.255.255.0
```

## Ongoing work
* Performance improvements (implementation of double loop)
* Full multi-thread support
* More test cases
* Support for moving endpoints (e.g. src IP change)
* Support for IP headers with options
* IPv6 support
Restart the Vagrant VM and VPP should see the interfaces when using `sudo vppctl sh int`
as `GigabitEthernet0/8/0` and `GigabitEthernet0/9/0`. Add the corresponding IPs:
```
set int state GigabitEthernet0/8/0 up
set int ip address GigabitEthernet0/8/0 192.168.100.2/24
set int state GigabitEthernet0/9/0 up
set int ip address GigabitEthernet0/9/0 192.168.101.2/24
```

## Known limitations
* Currently only support for 2048 concurrent flows (should be more than enough for initial tests)
## Measurement results

The VPP plugin writes latency measurement results to the `/tmp` folder using different
files for QUIC, TCP and PLUS traffic (`/tmp/latency_{plus,tcp,quic}_printf.out`).
The data is saved as CSV files. All latency estimations are in seconds.

### QUIC latency measurements
Header of the CSV file: `time,pn,host,spin_data,spin_new,pn_spin_data,pn_spin_new,vec_data,vec_new,heur_data,heur_new`
- `time`: time since start of VPP in seconds
- `pn`: packet number of observed QUIC packet
- `host`: server or client direction
- `spin_data`: latency estimation taking only the latency spin bit into account
- `spin_new`: does the `spin_data` contain a new estimation (0 or 1)
- `pn_spin_data`: latency estimation based on the spin bit only but rejecting reordered packets based on the packet number
- `pn_spin_new`: does the `pn_spin_new` contain a new estimation (0 or 1)
- `vec_data`: latency estimation based on the full spin signal (spin bit and VEC)
- `vec_new`: does the `vec_data` contain a new estimation (0 or 1)
- `heur_data`: latency estimation based on the spin bit only but rejecting RTT samples based on a heuristic
- `heur_new`: does the `heur_data` contain a new estimation (0 or 1)

More information can be found in our [IMC paper](https://nsg.ee.ethz.ch/fileadmin/user_upload/spinbit.pdf).

### TCP latency measurements
Header of the CSV file: `time,host,seq_num,vec_data,vec_new,single_ts_rtt_data,single_ts_rtt_new,all_ts_rtt_data,all_ts_rtt_new,vec_ne_zero_data,vec_ne_zero_new`
- `time`: time since start of VPP in seconds
- `host`: server or client direction
- `seq_num`: sequence number of observed TCP packet
- `vec_data`: latency estimation based on the full spin signal (spin bit and VEC)
- `vec_new`: does the `vec_data` contain a new estimation (0 or 1)
- `single_ts_rtt_data`: latency estimation based on one timestamp per RTT
- `single_ts_rtt_new`: does the `single_ts_rtt_data` contain a new estimation (0 or 1)
- `all_ts_rtt_data`: latency estimation based on every available timestamp value
- `all_ts_rtt_new`: does the `all_ts_rtt_data` contain a new estimation (0 or 1)
- `vec_ne_zero_data`: latency estimation based on the full spin signal (spin bit and VEC) taking every non-zero VEC value into account
- `vec_ne_zero_new`: does the `vec_ne_zero_data` contain a new estimation (0 or 1)

### PLUS latency measurements
Header of the CSV file: `time,host,#pkt,psn,pse,cat,psn_pse_data,psn_pse_new`
- `time`: time since start of VPP in seconds
- `host`: server of client directions
- `#pkt`: number of packet in PLUS flow
- `psn`: Packet Serial Number of the observed PLUS packet
- `pse`: Packet Serial Echo of the observed PLUS packet
- `cat`: Connection and Association Token of the observed PLUS packet
- `psn_pse_data`: latency estimation based on the PSN and PSE
- `psn_pse_new`: does the `psn_pse_data` contain a new estimation (0 or 1)

More information can be found in our [PLUS paper](https://nsg.ee.ethz.ch/fileadmin/user_upload/CNSM_2017.pdf).
2 changes: 1 addition & 1 deletion plus-plugin/Makefile.am → latency-plugin/Makefile.am
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ ACLOCAL_AMFLAGS = -I m4
vppapitestpluginsdir = ${libdir}/vpp_api_test_plugins
vpppluginsdir = ${libdir}/vpp_plugins

include plus.am
include latency.am

%.api.h: %.api
mkdir -p `dirname $@` ; \
Expand Down
2 changes: 1 addition & 1 deletion plus-plugin/configure.ac → latency-plugin/configure.ac
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
AC_INIT(vpp_plugins, 1.0)
AC_INIT(latency_plugins, 1.0)
LT_INIT
AM_INIT_AUTOMAKE
AM_SILENT_RULES([yes])
Expand Down
22 changes: 11 additions & 11 deletions plus-plugin/plus.am → latency-plugin/latency.am
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,21 +11,21 @@
# See the License for the specific language governing permissions and
# limitations under the License.

vppapitestplugins_LTLIBRARIES += plus_test_plugin.la
vppplugins_LTLIBRARIES += plus_plugin.la
vppapitestplugins_LTLIBRARIES += latency_test_plugin.la
vppplugins_LTLIBRARIES += latency_plugin.la

plus_plugin_la_SOURCES = \
plus/plus.c \
plus/node.c \
plus/plus_plugin.api.h
latency_plugin_la_SOURCES = \
latency/latency.c \
latency/node.c \
latency/latency_plugin.api.h

API_FILES += plus/plus.api
API_FILES += latency/latency.api

nobase_apiinclude_HEADERS += \
plus/plus_all_api_h.h \
plus/plus_msg_enum.h \
plus/plus.api.h
latency/latency_all_api_h.h \
latency/latency_msg_enum.h \
latency/latency.api.h

plus_test_plugin_la_SOURCES = plus/plus_test.c plus/plus_plugin.api.h
latency_test_plugin_la_SOURCES = latency/latency_test.c latency/latency_plugin.api.h

# vi:syntax=automake
2 changes: 1 addition & 1 deletion plus-plugin/plus/plus.api → latency-plugin/latency/latency.api
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

/* Define a simple binary API to control the feature */

autoreply define plus_enable_disable {
autoreply define latency_enable_disable {
/* Client identifier, set from api_main.my_client_index */
u32 client_index;

Expand Down
Loading

0 comments on commit 163cdd3

Please sign in to comment.