diff --git a/test/README.md b/test/README.md index c8ff94c4257e..6aaff56d2f1c 100644 --- a/test/README.md +++ b/test/README.md @@ -1,31 +1,34 @@ -# Project structure +# Mullvad VPN end to end test framework -## test-manager +## Project structure + +### test-manager The client part of the testing environment. This program runs on the host and connects over a virtual serial port to the `test-runner`. The tests themselves are defined in this package, using the interface provided by `test-runner`. -## test-runner +### test-runner The server part of the testing environment. This program runs in guest VMs and provides the `test-manager` with the building blocks (RPCs) needed to create tests. -## test-rpc +### test-rpc A support library for the other two packages. Defines an RPC interface, transports, shared types, etc. -# Prerequisities + +## Prerequisites For macOS, the host machine must be macOS. All other platforms assume that the host is Linux. -## All platforms +### All platforms * Get the latest stable Rust from https://rustup.rs/. -## macOS +### macOS Normally, you would use Tart here. It can be installed with Homebrew. You'll also need `wireguard-tools`, a protobuf compiler, and OpenSSL: @@ -34,7 +37,7 @@ Normally, you would use Tart here. It can be installed with Homebrew. You'll als brew install cirruslabs/cli/tart wireguard-tools pkg-config openssl protobuf ``` -### Wireshark +#### Wireshark Wireshark is also required. More specifically, you'll need `wireshark-chmodbpf`, which can be found in the Wireshark installer here: https://www.wireshark.org/download.html @@ -47,7 +50,7 @@ dseditgroup -o edit -a THISUSER -t user access_bpf This lets us monitor traffic on network interfaces without root access. -## Linux +### Linux For running tests on Linux and Windows guests, you will need these tools and libraries: @@ -58,7 +61,51 @@ dnf install git gcc protobuf-devel libpcap-devel qemu \ wireguard-tools ``` -# Building the test runner +## Setting up testing environment + +First you need to build the images for running tests on, see [`BUILD_OS_IMAGE.md`](./docs/BUILD_OS_IMAGE.md). The `test-manager` then needs to be configured to use the image. + +Here is an example of how to create a new OS configuration for Linux and macOS: + +### Linux + +```bash +# Create or edit configuration +# The image is assumed to contain a test runner service set up as described in ./docs/BUILD_OS_IMAGE.md +cargo run --bin test-manager set debian11 qemu ./os-images/debian11.qcow2 linux \ + --package-type deb --architecture x64 \ + --provisioner ssh --ssh-user test --ssh-password test + +# Try it out to see if it works - you should reach the VM's graphical desktop environment +cargo run --bin test-manager run-vm debian11 +``` + +### macOS + + +```bash +# Download some VM image +tart clone ghcr.io/cirruslabs/macos-ventura-base:latest ventura-base + +# Create or edit configuration +# Use SSH to deploy the test runner since the image doesn't contain a runner +cargo run --bin test-manager set macos-ventura tart ventura-base macos \ + --architecture aarch64 \ + --provisioner ssh --ssh-user admin --ssh-password admin + +# Try it out to see if it works +cargo run -p test-manager run-vm macos-ventura +``` + +## Testing the app + +To automatically download and test a pre-built version of the app, use the `test-by-version.sh` script, see `test-by-version.sh --help` for instructions. + +To manually invoke `test-manager`, start by checking out the desired git version of this repo. Next, [build the app](../BuildInstructions.md) for the target platform then build the GUI test binary using `$(cd ../gui && npm run build-test-executable)`. The newly built packages will be located in the `../dist` folder by default. + +Next: build the `test-runner` + +### Building the test runner Building the `test-runner` binary is done with the `build-runner.sh` script. Currently, only `x86_64` platforms are supported for Windows/Linux and `ARM64` (Apple Silicon) for macOS. @@ -69,76 +116,48 @@ For example, building `test-runner` for Windows would look like this: ./scripts/container-run.sh ./scripts/build-runner.sh windows ``` -## Linux -Using `podman` is the recommended way to build the `test-runner`. See the [Linux section under Prerequisities](#Prerequisities) for more details. +#### Linux +Using `podman` is the recommended way to build the `test-runner`. See the [Linux section under Prerequisities](#prerequisites) for more details. ``` bash ./scripts/container-run.sh ./scripts/build-runner.sh linux ``` -## macOS +#### macOS ``` bash ./scripts/build-runner.sh macos ``` -## Windows +#### Windows The `test-runner` binary for Windows may be cross-compiled from a Linux host. ``` bash ./scripts/container-run.sh ./scripts/build-runner.sh windows ``` -# Building base images - -See [`BUILD_OS_IMAGE.md`](./docs/BUILD_OS_IMAGE.md) for how to build images for running tests on. - -# Running tests +### Running the tests -See `cargo run --bin test-manager` for details. +After configuring the VM image using `test-manager set` and building the required packages (see [previous step](#setting-up-testing-environment)), `test-manager run-tests` is used to launch the tests. See `cargo run --bin test-manager -- run-tests --help` for details. -## Linux +Here is an example of how to run all tests using the Linux/macOS VM we set up earlier: -Here is an example of how to create a new OS configuration and then run all tests: +#### Linux ```bash -# Create or edit configuration -# The image is assumed to contain a test runner service set up as described in ./docs/BUILD_OS_IMAGE.md -cargo run --bin test-manager set debian11 qemu ./os-images/debian11.qcow2 linux \ - --package-type deb --architecture x64 \ - --provisioner ssh --ssh-user test --ssh-password test - -# Try it out to see if it works - you should reach the VM's graphical desktop environment -cargo run --bin test-manager run-vm debian11 - # Run all tests -cargo run --bin test-manager run-tests debian11 \ +cargo run --bin test-manager run-tests --vm debian11 \ --display \ --account 0123456789 \ --app-package \ --app-package-to-upgrade-from 2023.2 ``` -## macOS - -Here is an example of how to create a new OS configuration (on Apple Silicon) and then run all -tests: +#### macOS ```bash -# Download some VM image -tart clone ghcr.io/cirruslabs/macos-ventura-base:latest ventura-base - -# Create or edit configuration -# Use SSH to deploy the test runner since the image doesn't contain a runner -cargo run --bin test-manager set macos-ventura tart ventura-base macos \ - --architecture aarch64 \ - --provisioner ssh --ssh-user admin --ssh-password admin - -# Try it out to see if it works -#cargo run -p test-manager run-vm macos-ventura - # Run all tests -cargo run --bin test-manager run-tests macos-ventura \ +cargo run --bin test-manager run-tests --vm macos-ventura \ --display \ --account 0123456789 \ --app-package \ @@ -147,5 +166,4 @@ cargo run --bin test-manager run-tests macos-ventura \ ## Note on `scripts/ci-runtests.sh` -Account tokens are read (newline-delimited) from the path specified by the environment variable -`ACCOUNT_TOKENS`. Round robin is used to select an account for each VM. +`scripts/ci-runtests.sh` is the script that GitHub actions uses to invokes the `test-manager`, with similar functionality as `test-by-version.sh`. Note that account tokens are read (newline-delimited) from the path specified by the environment variable `ACCOUNT_TOKENS`. Round robin is used to select an account for each VM.