Finch Common End-to-end Tests

This repository contains common e2e tests that can be imported as Go functions (The Functions).

The scope of this repository is limited to container development commands (i.e., no tests for VM lifecycle).

To enable users of this repository to freely extend the set of tests to be run, this repository is not in charge of the lifecycle of the testing environment (e.g., spin up or tear down the testing VM):

• When The Functions are called, it is assumed that the environment for the specified test subject is already set up (e.g., assume that finch is the test subject here, the underlying VM is up, and sanity-checking command like finch version can return successfully).
• The Functions may remove any existing objects in the testing environment (e.g., a container that is currently running may be stopped and removed by The Functions).
• The Functions will remove any objects generated by it (e.g., an image pulled during the testing).

Note that The Functions may create temporary files under the home directory of the current user, so make sure that the home directory is writable by the "subject" (see New in option/option.go for more details), or the tests may fail. For more details, see ffs/create.go.

Development Flow

1. Loop
   1. Update the tests as you wish.
   2. To save time, use the Focus decorator to only run the tests you're developing.
   3. Run make run (you may need to tune the Makefile variables according to your setup). If the output makes sense to you, exit the loop. If not, go back to the step 1.
2. Remove the Focus decorators added during development.
3. Create a pull request.
4. After the pull request is merged, cut a release if necessary. Please refer to the versioning section if you need to cut a release.

Versioning

We use semantic versioning. There are some concrete examples in the sections below. They are meant to be extended when new interesting cases are discovered so that in the future, we can speed up the process of deciding which version to bump by having some references here.

Major

• Change the signature of option.New to accept more arguments in a backward-incompatible manner.

Minor

• Add a new test (i.e., add an exported function to the common-tests package).

Patch

• Change the image that is pulled in tests.Pull from alpine to a smaller image.

Sync Between Tests and Code

This section explains the workflow to keep tests up to date along with code changes.

Take Finch CLI as an example, when adding/updating an e2e test, it is likely that the publicly available Finch CLI does not have the corresponding functionality yet. It is because the functionality cannot be added to Finch CLI until the corresponding test is in place. In other words, the test has to be merged into Finch Test, and a release will be cut to include the newly added test, then the test can be used by Finch CLI via referencing the newly created release in its go.mod. Last, the CI of Finch CLI can run with the updated test and the added functionality.

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.