Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
@Serock3 @MarkusPettersson98
Serock3 authored and MarkusPettersson98 committed Aug 9, 2024
1 parent db134ab commit c29f619
Showing 1 changed file with 69 additions and 51 deletions.
120 changes: 69 additions & 51 deletions test/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,31 +1,34 @@
# Project structure
# Mullvad VPN end to end test framework

## test-manager
## Project structure

### test-manager

The client part of the testing environment. This program runs on the host and connects over a
virtual serial port to the `test-runner`.

The tests themselves are defined in this package, using the interface provided by `test-runner`.

## test-runner
### test-runner

The server part of the testing environment. This program runs in guest VMs and provides the
`test-manager` with the building blocks (RPCs) needed to create tests.

## test-rpc
### test-rpc

A support library for the other two packages. Defines an RPC interface, transports, shared types,
etc.

# Prerequisities

## Prerequisites

For macOS, the host machine must be macOS. All other platforms assume that the host is Linux.

## All platforms
### All platforms

* Get the latest stable Rust from https://rustup.rs/.

## macOS
### macOS

Normally, you would use Tart here. It can be installed with Homebrew. You'll also need
`wireguard-tools`, a protobuf compiler, and OpenSSL:
Expand All @@ -34,7 +37,7 @@ Normally, you would use Tart here. It can be installed with Homebrew. You'll als
brew install cirruslabs/cli/tart wireguard-tools pkg-config openssl protobuf
```

### Wireshark
#### Wireshark

Wireshark is also required. More specifically, you'll need `wireshark-chmodbpf`, which can be found
in the Wireshark installer here: https://www.wireshark.org/download.html
Expand All @@ -47,7 +50,7 @@ dseditgroup -o edit -a THISUSER -t user access_bpf

This lets us monitor traffic on network interfaces without root access.

## Linux
### Linux

For running tests on Linux and Windows guests, you will need these tools and libraries:

Expand All @@ -58,7 +61,51 @@ dnf install git gcc protobuf-devel libpcap-devel qemu \
wireguard-tools
```

# Building the test runner
## Setting up testing environment

First you need to build the images for running tests on, see [`BUILD_OS_IMAGE.md`](./docs/BUILD_OS_IMAGE.md). The `test-manager` then needs to be configured to use the image.

Here is an example of how to create a new OS configuration for Linux and macOS:

### Linux

```bash
# Create or edit configuration
# The image is assumed to contain a test runner service set up as described in ./docs/BUILD_OS_IMAGE.md
cargo run --bin test-manager set debian11 qemu ./os-images/debian11.qcow2 linux \
--package-type deb --architecture x64 \
--provisioner ssh --ssh-user test --ssh-password test

# Try it out to see if it works - you should reach the VM's graphical desktop environment
cargo run --bin test-manager run-vm debian11
```

### macOS


```bash
# Download some VM image
tart clone ghcr.io/cirruslabs/macos-ventura-base:latest ventura-base

# Create or edit configuration
# Use SSH to deploy the test runner since the image doesn't contain a runner
cargo run --bin test-manager set macos-ventura tart ventura-base macos \
--architecture aarch64 \
--provisioner ssh --ssh-user admin --ssh-password admin

# Try it out to see if it works
cargo run -p test-manager run-vm macos-ventura
```

## Testing the app

To automatically download and test a pre-built version of the app, use the `test-by-version.sh` script, see `test-by-version.sh --help` for instructions.

To manually invoke `test-manager`, start by checking out the desired git version of this repo. Next, [build the app](../BuildInstructions.md) for the target platform then build the GUI test binary using `$(cd ../gui && npm run build-test-executable)`. The newly built packages will be located in the `../dist` folder by default.

Next: build the `test-runner`

### Building the test runner

Building the `test-runner` binary is done with the `build-runner.sh` script.
Currently, only `x86_64` platforms are supported for Windows/Linux and `ARM64` (Apple Silicon) for macOS.
Expand All @@ -69,76 +116,48 @@ For example, building `test-runner` for Windows would look like this:
./scripts/container-run.sh ./scripts/build-runner.sh windows
```

## Linux
Using `podman` is the recommended way to build the `test-runner`. See the [Linux section under Prerequisities](#Prerequisities) for more details.
#### Linux
Using `podman` is the recommended way to build the `test-runner`. See the [Linux section under Prerequisities](#prerequisites) for more details.

``` bash
./scripts/container-run.sh ./scripts/build-runner.sh linux
```

## macOS
#### macOS

``` bash
./scripts/build-runner.sh macos
```

## Windows
#### Windows
The `test-runner` binary for Windows may be cross-compiled from a Linux host.

``` bash
./scripts/container-run.sh ./scripts/build-runner.sh windows
```

# Building base images

See [`BUILD_OS_IMAGE.md`](./docs/BUILD_OS_IMAGE.md) for how to build images for running tests on.

# Running tests
### Running the tests

See `cargo run --bin test-manager` for details.
After configuring the VM image using `test-manager set` and building the required packages (see [previous step](#setting-up-testing-environment)), `test-manager run-tests` is used to launch the tests. See `cargo run --bin test-manager -- run-tests --help` for details.

## Linux
Here is an example of how to run all tests using the Linux/macOS VM we set up earlier:

Here is an example of how to create a new OS configuration and then run all tests:
#### Linux

```bash
# Create or edit configuration
# The image is assumed to contain a test runner service set up as described in ./docs/BUILD_OS_IMAGE.md
cargo run --bin test-manager set debian11 qemu ./os-images/debian11.qcow2 linux \
--package-type deb --architecture x64 \
--provisioner ssh --ssh-user test --ssh-password test

# Try it out to see if it works - you should reach the VM's graphical desktop environment
cargo run --bin test-manager run-vm debian11

# Run all tests
cargo run --bin test-manager run-tests debian11 \
cargo run --bin test-manager run-tests --vm debian11 \
--display \
--account 0123456789 \
--app-package <git hash or tag> \
--app-package-to-upgrade-from 2023.2
```

## macOS

Here is an example of how to create a new OS configuration (on Apple Silicon) and then run all
tests:
#### macOS

```bash
# Download some VM image
tart clone ghcr.io/cirruslabs/macos-ventura-base:latest ventura-base

# Create or edit configuration
# Use SSH to deploy the test runner since the image doesn't contain a runner
cargo run --bin test-manager set macos-ventura tart ventura-base macos \
--architecture aarch64 \
--provisioner ssh --ssh-user admin --ssh-password admin

# Try it out to see if it works
#cargo run -p test-manager run-vm macos-ventura

# Run all tests
cargo run --bin test-manager run-tests macos-ventura \
cargo run --bin test-manager run-tests --vm macos-ventura \
--display \
--account 0123456789 \
--app-package <git hash or tag> \
Expand All @@ -147,5 +166,4 @@ cargo run --bin test-manager run-tests macos-ventura \

## Note on `scripts/ci-runtests.sh`

Account tokens are read (newline-delimited) from the path specified by the environment variable
`ACCOUNT_TOKENS`. Round robin is used to select an account for each VM.
`scripts/ci-runtests.sh` is the script that GitHub actions uses to invokes the `test-manager`, with similar functionality as `test-by-version.sh`. Note that account tokens are read (newline-delimited) from the path specified by the environment variable `ACCOUNT_TOKENS`. Round robin is used to select an account for each VM.

0 comments on commit c29f619

Please sign in to comment.